Using Management Science and Engineering to Improve Cancer Care Delivery
Prepared by Karen Sepucha for the National Cancer Institute November 11, 2010
Executive Summary ........................................................................................................................ 1 Strategy Paper ................................................................................................................................. 3 Recommendations: .................................................................................................................... 11 Appendix A: Using Management Science and Engineering to Improve Cancer Care Delivery Working Group ............................................................................................................................. 17 A.1 Working Group Agenda ..................................................................................................... 17 A.2 Participant List ................................................................................................................... 19 A.3 Working group summary ................................................................................................... 20 Appendix B. Key Informant Interviews........................................................................................ 42 Appendix C. Additional background materials for working group .............................................. 45 Appendix D: Selected initiatives using engineering across NIH .................................................. 56 References ..................................................................................................................................... 60
The report summarizes our findings about the potential for management, science, and engineering (MS&E) to advance health systems interventions and evaluation research for cancer care. Our definition of management, science and engineering encompasses a broad set of disciplines, from traditional engineering fields such as operations research, industrial engineering and systems engineering, as well as social sciences. The quantitative rigor and analytic methods from the engineering disciplines can help provide insight into complex systems, but this insight needs to be balanced by attention to the organizational habits, culture and behavior change that is required for successful implementation of systems change from the social sciences. The purpose was to generate a strategy for NCI to build its research program in cancer care delivery from diagnosis to end-of-life, and to identify methods and tools from MS&E that might be relevant.
We started by drafting a strategy paper that outlined the growing need for attention to the organization and delivery of cancer care as a research issue. We also described the potential relevance of MS&E approaches, methods and tools for cancer care. Next, we conducted interviews with key informants in the cancer community to identify the most pressing challenges from their perspectives. With this information, we refined the strategy paper and focused on some promising MS&E methods and tools that addressed the most pressing clinical issues. Then, we invited a small group of clinical experts and researchers from management science and engineering to participate in an interactive working group. They were all sent a draft of the strategy paper as background material. During the one day session, participants shared their expertise, provided their advice and feedback on the strategy paper, and generated a list of high priority research questions and next steps required to advance this agenda.
In this report we include the revised strategy paper, selected excerpts from the interviews, as well as an edited summary of the working group. There were several key findings that are worth highlighting here: 1
1. There is fairly little understanding of MS&E on the part of cancer community and vice versa, as such, any strategy needs to start with some awareness building. 2. There are several pressing challenges facing cancer clinicians and administrators that existing MS&E tools and methods could address, as such, there is the potential for some focused research projects (e.g. examining impact of relational coordination on cancer outcomes) that could be carried out in the short term. 3. MS&E spans academic disciplines (e.g. industrial engineering, management science, operations research) and schools (e.g. arts and sciences, business schools, engineering schools). Given the intersection of MS&E and cancer care delivery spans many disciplines, it is reasonable for several governmental agencies (e.g. NSF, NCI, AHRQ) to lead or initiate a collaborative endeavor. As such, any strategy will need to develop a strong collaboration among key stakeholders in order to ensure a strong, successful research program.
There are many expectations in the new health care reform bill regarding what organizations should be doing to try and help improve healthcare value; however, most of the focus is on primary care. Consequently, NCI is faced with two questions, how does specialty care factor into this debate? And what can research do to help guide these efforts? Management, science and engineering is an exciting approach that could be used to help build this program and provide information to achieve NCI’s objective of reducing death and suffering from cancer. This approach will be challenging though, since NCI’s current health services research is focused on individuals, and has rarely focused the cancer care delivery system. This project identified lots of enthusiasm for moving ahead, and highlights several opportunities for and challenges to using MS&E to help NCI with this endeavor.
The purpose of this paper is to identify opportunities for management, science and engineering (MS&E) to advance health systems interventions and evaluation research for cancer care. We have defined MS&E broadly, recognizing that the field spans several disciplines including but not limited to management science, operations research, industrial engineering, and industrial and systems engineering. Though each of these engineering fields has different methods and traditions, generally these researchers are concerned with the development of knowledge, tools, and methods required to make decisions and shape policies, to configure organizational structures and design systems, and to solve problems associated with the information-intensive technology based economy. We do not adopt an engineering-focused definition of MS&E, where the purpose is to give engineers a foundation in management (and vice versa) with the goal of ensuring that engineering companies can be more effective and successful. Rather, we wanted to apply approaches of MS&E to health care delivery, which is an information-intensive and service-oriented economy. The rigor and analytics from engineering that can help provide insight into complex systems needs to be balanced by attention to the organizational habits, culture and behavior change that is required for successful implementation of systems change. As a result, we adopt an even broader definition of MS&E that recognizes the importance of social sciences to understanding and intervening in complex, human systems. Here, we describe how MS&E methods, broadly defined, may help systems of cancer care delivery improve their ability to deliver high quality, high value, patient-centered care.
Trends in the delivery of cancer care The mission of the NCI is to eliminate death and suffering due to cancer. This goal will not be fully achieved until we can prevent all cancers from occurring. For the millions of Americans who will receive a cancer diagnosis before that ultimate goal is achieved, there is much more that we can do to reduce death from this disease and relieve the suffering caused by cancer and its treatment.
Advances in science and information technology have far outpaced the progress made in systems for delivery of health services. Molecular tests are now available that can analyze a breast cancer tumor and predict whether or not that breast cancer will recur. Giving doctors sophisticated tools to tailor treatments to the individual tumor will revolutionize the care of patients—enabling thousands of patients to safely forgo toxic treatments and providing those at high risk of dying from their cancer with more targeted and effective treatments. Equally, if not more exciting, is the promise of molecular tools to more accurately predict risk of getting cancer and which may ultimately lead to more effective prevention. A number of these tools are ready for use today, and many more are waiting to be developed and validated.
The potential gains from personalization will not be fully realized if we do not simultaneously prepare patients, providers and the health care system to handle the increased complexity in decision making and increased pressure for coordination and collaboration within and across teams. Several critical questions arise as we seek to realize the potential of personalized medicine. What are the best ways to organize and deliver care in this new paradigm? How can we harness the current knowledge and apply it more appropriately and effectively? How can we anticipate the future innovations and prepare the delivery system to adopt them? Are there ways to design the delivery system to facilitate translation of the discoveries from the bench to the bedside, promote enrollment in clinical trials and then to support continuous learning back to the bench?
Importance of change management: There are many barriers to implementing new tools appropriately in practice. Many innovations are disruptive—they require new processes of care and force the care delivery team to work together in new ways on new tasks. For example, partial breast irradiation techniques link radiation oncologists and surgeons in new ways, and neoadjvuant chemotherapy regimens reorganize the traditional order of treatment modalities. However, these opportunities are missed if patients never see a radiation oncologist or medical oncologist until after surgery. Increased personalization will also introduce more complexity into care delivery as infusion centers will be challenged to deliver more agents, in more combinations without compromising safety. Without
sufficient attention to optimizing the implementation of these advances, even effective tests and treatments may not live up to their potential, and may do more harm than good.
Some centers and organizations naturally adopt and deliver innovations better than others. Studies of health care organizations that do adapt and innovate have found some surprising results. The factors that many associate with delivery of state of the art care, such as the experience of clinicians, volume, or type of institution (e.g. academic), are not necessarily those that predict successful adoption of new techniques. For example, Pisano et al. studied the introduction of a new technique for cardiac surgery that required rethinking roles and procedures in the operating room. (Pisano et al., 2001) Organizations that introduced it as a “project,” specifically planned for the new roles and behaviors, and engaged the team to practice in a supportive environment, were most likely to adopt the new technology. Having a senior clinician on board, or having high volume of cases did not predict successful adoption.
Need for better decision support systems As we anticipate the innovations that may come, we also need to recognize how far we are from delivering care commensurate with the considerable evidence base that we already have. The directors of the NCI-designated cancer centers have recognized the importance of the opportunity to “substantially reduce the death from cancer just by broadening the application of the knowledge we have today” p. 5 (National Cancer Institute, 2006). Quality reports find large gaps in the implementation of interventions that we know work—and these gaps are not limited to cancer centers, but span the continuum from prevention and screening through treatment and survivorship. Many cancer patients do not receive guideline care. (Harlan et al., 2005) Only about half of Americans older than 50 have ever had screening for colon cancer, and fewer than 40% are up to date with their screening. (Phamm et al., 2005) It is not all underuse of care, but also overuse that is evident as many women receive mammograms, MRIs and other screening tests more often than recommended after breast cancer. (Keating et al., 2007) This focus on breast cancer surveillance often comes at the expense of other needed preventive activities (e.g. heart disease). (Earle et al., 2003)
Bridging different research disciplines to create the solutions The National Cancer Institute is uniquely positioned to play a vital role in advancing a research agenda to examine how the delivery of cancer care can be designed to improve patient outcomes. NCI has a huge influence on the delivery of care, through its support of the NCI-designated cancer centers, Community Clinical Oncology Group (CCOP), Cancer Research Network (CRN), NCI-sponsored Cooperative Clinical Trials Groups as well as through its ability to influence practitioners. To do this well will require reaching out across federal agencies, to AHRQ, CMS, VA, DOD and others who influence the delivery of care. Table 1 summarizes some of the key organizations that should be engaged in these efforts. The list is not exhaustive but highlights some of the critical organizations, and types of stakeholders that need to be engaged. Many of these agencies are asking similar questions, (e.g. how to improve health care delivery and what role might health care engineering play?) Though it is unlikely that one or even a few pilots may serve all interests, it would be valuable to develop a collaboration around the exploration of MS & E’s impact upon healthcare delivery. Some amount of consensus on expected outcomes and metrics used in evaluation would be helpful in drawing conclusions from dissimilar research projects.
Table 1: List of some key organizations (alphabetical order) Agency for Healthcare Research and Quality American Cancer Society American Society of Clinical Oncology Center for Medicare and Medicaid Services Comprehensive Cancer Center Consortium for Quality Improvement (C4QI) Department of Defense Institute of Medicine INFORMS (Institute for Operations Research and Management Science) National Academy of Engineering National Cancer Institute (NCI)
NCI designated comprehensive cancer centers National Coalition of Cancer Survivors NCI Community Cancer Centers Program National Comprehensive Cancer Network National Institutes of Health National Patient Safety Foundation National Science Foundation Oncology Nursing Society Quality of Cancer Care Committee Veteran’s Administration
The NCI can use its convening power to engage experts in management, science and engineering (MS&E), a discipline that focuses on the organization and delivery of products and services, and the behaviors of workers across all industries. While epidemiology studies typically focus on correlates of disease, they usually ignore upstream influences. In the case of hypertension, issues such as poverty, racism, and unemployment have been found to play a large role, yet these are usually ignored in most medical studies. (McGuire, 2005) Clinical studies also tend to ignore the impact of downstream differences in delivery of interventions on outcomes. Studies have exposed a link between volume and surgical outcomes, and between the culture of the care team and outcomes. (Shrag et al 2000 and Gittel et al 2001) The participants in the delivery team, the presence or absence of clinical decision support systems, infrastructure support, and the reimbursement policies may all play an important role in understanding patient outcomes. Management science and engineering takes many of these issues as the core focus to develop an understanding of how the organization and delivery of care, the patient context and environment, and other non-clinical factors may contribute or detract from the objectives of health and other outcomes.
Traditionally, MS&E researchers have focused their efforts in manufacturing and product industries, but recently the discipline has moved into the service sector. The first applications have focused in telecommunications, transportation, finance and retail. These sectors have large information technology infrastructures and have extensive data that can be mined to drive the strategic and tactical decisions made by managers and workers. The application of rigorous analytic techniques such as statistical process control, queuing theory, modeling and simulation, has resulted in significant efficiency and quality improvements in these industries.
Given the increasingly significant role the healthcare industry has on the U.S. economy, researchers from management science are turning their attention to this industry. A recent report jointly published by the Institute of Medicine and the National Academy of Engineering titled, “Building a Better Delivery System: A New Engineering/Health Care Partnership” (2005) outlines several recommendations for the different ways engineering may address some of the system failures in health care. Some key areas include the design of financial and reimbursement
systems, supply chain management, and the development of information systems and knowledge management.
The report describes several methods that can be used to aid in the design, analysis and control of health care systems. Several have been applied already in health care. For example, statistical process control and queuing theory can be used to help with operating room scheduling or to eliminate bottlenecks with patient flow in a system. Modeling and simulation can be used to examine outbreaks of disease and corresponding demand on hospitals. Human-factors engineering (which focuses on how humans interact with each other, the technology and the environment to accomplish a task, and then redesigns the process to minimize errors) can be used to make sure instructions are understandable to patients, or to examine whether electronic medical records enable continuity of care. In addition, other methods such as supply-chain management, financial engineering and risk analysis, and knowledge management are among the tools that have been identified to have immediate potential for adaptation to health care delivery.
Increasing efficiency of an automated assembly line is not the same as increasing the efficiency of a cancer center. Even the best analytic tools will fail if the members of the system do not actively support their use. As we have defined it, MS&E is not only the quantitative fields, but also draws from social and behavioral sciences in order to understand and model the human aspect of the health care delivery.
Some management science researchers have looked to health care as a model for generating knowledge about effectiveness of management practices. A series of recent articles and books promoting “evidence-based management” charges business managers to base their decisions on strong evidence, as doctors do in evidenced-based medicine. (Pfeffer and Sutton, 2006; Shortell et al., 2007) The three core principles of evidence-based management are familiar (1) Demand evidence: ensure that organizations have information to assess operations and performance and commitment to use metrics to judge performance, even when data not available. For example, health care providers should know their own complication rates, and how they compare to local, regional and national rates. Few surgeons practicing in the U.S. today would be able to access those figures easily. (2) Establish culture that reinforces speaking truth about how things are 8
going, (this is in keeping with the culture shift that has been emphasized by the patient safety movement). (3) Examine logic behind every decision and recommendation, and always examine assumptions.
The insight for medicine is that the same rigor and research that is applied to evaluating new treatments should be applied to the organization and delivery of care. For cancer care this may lead to examination of different issues such as, does multidisciplinary team-based care in cancer enable medical advances to be adopted sooner, or does it just waste doctors’ time? Also, as we have more and more cancer survivors living longer, how should the cancer care delivery system balance the needs of the newly diagnosed with those of survivors?
There are several places where management science and engineering techniques are being applied to medicine already. These applications are occurring at different levels of the health care system and across different clinical situations. For example, the Institute for Healthcare Improvement has conducted over 50 learning collaboratives that bring hospitals and groups together that are committed to address a gap between what we know and what we do. These groups have published some remarkable findings such as, reducing waiting times by half, reducing hospital readmissions for patients with congestive heart failure by half, and reducing worker absenteeism by 25%. (IHI 2003) This approach, or others, such as continuous quality improvement or lean production techniques, could be applied to issues around the quality of cancer care delivery.
Another example is Intermountain Healthcare which has developed a sophisticated approach to ensuring delivery of effective care. They have created teams to design and maintain standard protocols for certain common diagnostic and treatment situations. These are standing teams who periodically review and update the protocols in order to ensure that they are relevant, accurate and efficient both for patient outcomes and organizational performance. (Bohmer, 2010)
Anticipating and preparing for future changes
The recent health care legislation has generated higher scrutiny of the health care delivery system. The push for accountable health care organizations (ACOs) highlights the increasing need to support coordination and cooperation among physicians and hospitals—as well as the need for more attention to different models for organization and delivery. The hope is that the ACOs will eliminate some of the perverse incentives that have led to overuse and waste, while improving quality and outcomes. In order to realize the vision of a high quality, high value health care sytem, it will require experimentation. Atul Gawande’s article in the New Yorker is talking about accountable care organizations, but sums up what we hope to do in cancer care delivery “Accountable health care organizations… will, by necessity, be an experiment. We will need to do in-depth research on what makes the best systems successful—the peer-review committees? recruiting more primary-care doctors and nurses? putting doctors on salary?—and disseminate what we learn. Congress has provided vital funding for research that compares the effectiveness of different treatments, and this should help reduce uncertainty about which treatments are best. But we also need to fund research that compares the effectiveness of different systems of care— to reduce our uncertainty about which systems work best for communities. These are empirical, not ideological, questions.” – Atul Gawande New Yorker 2009
There is clearly a question about where specialty care in general, and cancer care in particular, fall in the current debate. It is also important to ask what role research might play to help guide these efforts? The key experts that we interviewed identified several challenges with delivery of health care that would benefit from closer examination. While all valued multidisciplinary care, individuals had mixed feelings about how that often gets implemented, e.g. the multidisciplinary conferences to evaluate patients. At some centers, cancer patients are seen by several doctors who then meet to discuss the case and return a recommendation to the patient. This kind of set up limits the number of patients that can be evaluated in a day, requires significant coordination among the providers’ schedules, and often requires patients to stay for four or more hours. At other centers, the patients only see one doctor and get a recommendation. Some of these centers discuss cases with a multidisciplinary group and others do not.
A few interviewees commented that these conferences were more for marketing than for patient care, whereas others saw them as critical part of delivery quality care. None were aware of any evidence of benefit in terms of improved patient care, nor were they aware of how to approach studying the impact of conferences on outcomes. Whether it is a multidisciplinary team conference or an ACO, the need to develop and test new ways of delivering care is needed. Practitioners are well-suited to identify the problems, but are not well-equipped to devise and test the solutions. We need to examine the resources and support needed to intervene, test new models of care and adopt best practice in delivery of cancer care. Management, science and engineering disciplines provide tested approaches that can be applied to more fully examine the system properties of cancer organizations and the implications for systems design and redesign to enable cancer care delivery organizations to achieve the patient centered principles outlined by the IOM.
Recommendations: Here we summarize the comments from the working group and interviews that led us to the three key insights from this work and then make some recommendations for how to move forward for each one.
1. There is fairly little understanding of MS&E on the part of cancer community and vice versa, as such, any strategy needs to start with some awareness building. We found that there is genuine interest in and awareness of a “problem” with delivery of cancer care on the part of the cancer clinicians and administrators. Most clinicians recognized the need for more resources, different incentives and better infrastructure and systems that support high quality, coordinated cancer care. However, despite the recognition of a problem, there was little evidence of any coordinated, evidence-based approach to address these challenges. There is not a means of identifying best practices across cancer centers, and there was no mention, among those we talked with, of experimentation or research on the organization and delivery of cancer care. There was also little understanding of management science and engineering, what that discipline covers, and how MS&E tools and approaches might help cancer care delivery.
Similarly, when talking with the MS&E researchers, many recognized the challenges inherent in the delivery of health care more broadly. Those that had experience in health care mainly worked in chronic disease management, primary care and prevention, or more traditional engineering type areas (e.g. optimization of OR scheduling). Few had experience in cancer care delivery or had a sense for the challenges specific to cancer care. This leads to the following recommended next steps: 1.a. The awareness building could take the form of a series of working groups that bring together key leaders from the clinical, business as well as the MS&E community. 1.b. Another suggested approach to awareness building could be accomplished through commissioning (and then publishing) a study that would examine what is happening in the delivery environment and identify larger scale systems changes (e.g. centralization of oncology practices, changes in marketplace, implications of health care reform) that are relevant for cancer centers. NCI could provide a valuable role to help examine what cancer centers might do now to take advantage of or excel in the changing environment and find leverage for increasing value? 1.c. Another powerful approach to awareness building is through experience, and to this end, the implementation of pilot(s) to trial MS&E applications to improve cancer care delivery would be important. The pilots should address an area of critical importance to cancer care, be sufficiently narrow in scope, with clear expectations, that the results can be readily observed, and these need to be conducted with a group that has been well prepared for the challenges of this type of multi-disciplinary, collaborative research.
2. There are several pressing challenges facing cancer clinicians and administrators that existing MS&E tools and methods could address, as such, there is the potential in the short term for some focused research projects. One of the key organizational issues that came up in the key informant interviews was the move toward multidisciplinary care in cancer. There was broad agreement with the importance of collaboration across specialties, however there were varied reactions to the effectiveness of some
of the ways organizations have approached this (e.g. multidisciplinary patient visits, team conferences). We made this issue a focus in the working group and had Dr. Salner (a clinician and administrator) discuss the issues from his perspective and then heard from two MS&E researchers, Dr. Gittell and Dr. Miller, on their approaches to studying and improving coordination in healthcare systems. The group questioned the level of evidence to support that multidisciplinary care is necessarily better than other approaches to cancer care. They wanted research to elucidate the mechanisms by which multidisciplinary approaches achieve outcomes, to provide evidence about the effect it has on organizational and patient outcomes. A recurring theme in the working group was the recognition of the inadequacy of current outcome measures, and the limitation of the focus on patient mortality. The group emphasized the importance of the development of metrics to capture issues like patient reported outcomes and patient experience, process of care, care coordination and continuity of care (e.g. relational coordination), organization performance (length of stay and time to treatment). Participants emphasized the importance of research to develop a “performance index” for cancer care and also noted that it is critical to connect with health IT meaningful use criteria to determine the core data set that we need to capture to evaluate performance of cancer care delivery (including patient reported outcomes). Several key research questions were identified during the working group which leads to the following set of recommended next steps:
2.a. How do we measure quality of care from a systems perspective? There was considerable discussion around metrics, consensus around the inadequacy of the current outcome measures that focus exclusively on cancer outcomes of mortality and morbidity, and desire to expand scope and content to capture system performance. 2.b. Does multidisciplinary care result in higher quality, patient-centered care? This question generated considerable interest and participants discussed the potential for measuring relational coordination across carefully selected cancer centers (e.g. those with different levelsof or different features of multidisciplinary care) and linking it to outcomes. 2.c. What is the relationship between system level features of the delivery of care and patient outcomes? There was interest in expanding our understanding of those features that matter most and have the most impact on outcomes (broadly defined to include clinical outcomes 13
(delivery of effective care), financial outcomes (costs), service (efficiently, timeliness), and patient experience.
3. MS&E spans academic disciplines (e.g. industrial engineering, engineering management, operations research) and schools (e.g. business schools, engineering schools). Given the intersection of MS&E and cancer care delivery spans many disciplines, it is reasonable for several governmental agencies (e.g. NSF, NCI, AHRQ) to lead or initiate a collaborative endeavor. As such, any strategy will need to develop a strong collaboration among key stakeholders in order to ensure a strong, successful research program. There is growing interest across government agencies, and initiatives examining health care delivery, using engineering and other approaches have started to form. For example, the recent conference sponsored by AHRQ and NSF to examine industrial and systems engineering (ISE), with an emphasis on information technology, and its ability to transform health care. At the end of the workshop, the conclusion was that ISE is not able, in its current state, to provide breakthrough change in the health care system. They developed an extensive research agenda, with an ambitious plan to generate new knowledge that may lead to breakthrough change. The first recommendation was to establish a collaboration among Federal agencies and key organizations (they had 24 listed, though NCI was not one of them). Although several of the key collaborators that were identified in our working group in Table 1, many of the cancer organizations that are critical for delivery of cancer care were not included. The NIH Commons Fund launched a Health Economics program, motivated by the recent health care legislation. A request for applications (R21) was released in August 2010 that focuses on identifying structural and organizational factors associated with high and low cost organizations. This call would not likely generate the kind of research that would directly relate to using engineering to make improvements in cancer care, as it was fairly focused in its disciplinary frame of economics, with costs and efficiency as key outcomes (there was also a focus on primary care with accountable care organizations and medical home.) However, it is an example of a mechanism for cross cutting research within NIH focused on the delivery of care. There is also growing interest in academic centers to examine health care delivery and the centers that do this have forged new models for collaboration across disciplines and schools. For 14
example. the Regenstrief Center for Healthcare Engineering (RCHE) at Purdue resides in Purdue’s Discovery Park, an entity that operates several centers that each focus on an emerging technology or social issue. The RCHE does not have a formal affiliation with any one college or department within the university nor does it provide an academic home to any of the affiliated researchers or faculty. The center works to provide a rich, productive research environment for faculty and students to contribute to the mission. The newly formed Center for Health Care Delivery Science at Dartmouth is similarly a center that has connections to but does not reside within either the medical school, engineering school or school of arts and sciences. During the working group, Dr. Witz clearly articulated some of the challenges and benefits of this type of research, including the importance of forging strong collaborative relationships with healthcare leaders and providers to further the research. This leads to our final insight and recommendation:
3.a. Ensuring that NCI is represented in other groups will help to ensure that the needs of cancer clinicians are represented and that the issues are not solely focused on primary care. (For example, NCI was not listed as one of the 24 organizations that are critical to engage in collaboration in the AHRQ report). 3.b. Outreach to the engineering schools in order to encourage research to focus on cancer care systems would help stimulate learning and lead to stronger collaborations. As these schools are being developed and curriculum being set, ensuring cancer case studies and opportunities for research projects in cancer care are available is important. 3.c. Through the awareness building and other activities, NCI should seek to build relationships with other government agencies and key organizations around the common goal of improving the quality of care delivery. It will be critical to get leaders on board (e.g. directors of the cancer centers) as well as those who have the resources to ensure that this agenda can move forward.
In summary, it is clear that there is excitement and enthusiasm for this endeavor. In light of the recent health care reform legislation, it is clear that NCI should expand its efforts to build a research portfolio that will advance our understanding of how to improve the delivery of cancer
care. Management, science and engineering is an exciting approach that could be used to help build this program. This approach will be challenging though, since NCI’s current health services research is focused on individuals, and has rarely focused the cancer care delivery system. However, there are some concrete steps that have been outlined here that can be started now that will build NCI’s capacity to examine these issues and make strides in achieving its mission of reducing death and suffering due to cancer.
Appendix A: Using Management Science and Engineering to Improve Cancer Care Delivery Working Group A.1 Working Group Agenda
August 31, 2010 NOVA Research Company Bethesda, MD AGENDA 8:15–9:00 a.m.
Introduction and Goals for the Working Group – Steve Clauser Background and Overview of the Day – Karen Sepucha
Session 1: Systems-Minded Cancer Care ¥
Multidisciplinary Cancer Care – Andy Salner
Relational Coordination as a Means of Measuring the Quality of
Collaboration – Jody Hoffer Gittell ¥
Challenges in the Continuity and Coordinating of Care – Anne
10:45 a.m.–12:15 p.m. Session 2: Knowledge-Based Cancer Care ¥
Informed Choice at Dartmouth-Hitchcock Medical Center – Dale
Collins Vidal ¥
How to Design Effective Research Teams That Bridge
Disciplines – Steve Witz
Session 3: Patient-Centered Cancer Care
Meeting Patients’ Needs in the Community – Peter Eisenberg
Patient-Centered Communication – Neeraj Arora
Patient-Centered Design of Delivery Systems – Jim Conway
Small Groups Will Summarize Key Issues and Draft Some Recommendations For Each Area
Synthesis of Group’s Recommendations Wrap Up and Concluding Remarks – Karen Sepucha/Steve Clauser
A.2 Participant List
Participant List Neeraj Arora, Ph.D. Research Scientist/Program Director Outcomes Research Branch Applied Research Program National Cancer Institute Bethesda, MD 20892 Phone: 301-594-6653 E-mail: [email protected]
Jody Hoffer Gittell, Ph.D. Associate Professor Heller School for Social Policy and Management Brandeis University Waltham, MA 02454 Phone: 781-736-3680 E-mail: [email protected]
Steven Clauser, Ph.D. Chief Outcomes Research Branch Applied Research Program National Cancer Institute Bethesda, MD 20892 Phone: 301-451-4402 Fax: 301-435-3710 E-mail: [email protected]
Anne Miller, Ph.D. Assistant Professor School of Nursing Vanderbilt University Medical Center Nashville, TN 37240 Phone: 615-936-7399 Fax: 615-736-7373 E-mail: [email protected]
Dale Collins Vidal, M.D. Plastic Surgery Dartmouth-Hitchcock Medical Center Lebanon, NH 03756 Phone: 603-650-5148 Fax: 603-650-8456 E-mail: [email protected]
Irene Prabhu Das, Ph.D. Health Scientist Applied Research Program Outcomes Research Branch National Cancer Institute Bethesda, MD 20892 Phone: 301-451-5803 E-mail: [email protected]
Jim Conway, M.S. Adjunct Lecturer School of Public Health Harvard University Woburn, MA 01801 Phone: 617-460-9799 E-mail: [email protected]
Andrew Salner, M.D. Director Gray Cancer Center Hartford Hospital Hartford, CT 06102 Phone: 860-545-2852 Fax: 860-545-4079 E-mail: [email protected]
John Deeken, M.D. Regional Director for Medical Oncology Medstar Health Georgetown University Hospital Washington, DC 20007 Phone: 202-444-3958 Fax: 202-444-9429 E-mail: [email protected]
Karen Sepucha, Ph.D. Senior Scientist General Medicine Massachusetts General Hospital Boston, MA 02114 Phone: 617-724-3350 Fax: 617-726-4120 E-mail: [email protected]
Peter Eisenberg, M.D. Medical Director California Cancer Care Greenbrae, CA 94904 Phone: 415-925-5001 Fax: 415-462-0701 E-mail: [email protected]
Steve Witz, Ph.D. Director Regenstrief Center for Healthcare Engineering Purdue University West Lafayette, IN 47905 Phone: 765-496-8303 Fax: 765-494-3023 E-mail: [email protected]
A.3 Working group summary The following is an edited summary of the meeting, “Using Management Science and Engineering to Improve Cancer Care Delivery,” held on August 31, 2010.
Opening remarks by Steve Clauser, Chief Outcomes Research Branch, NCI
The purpose of this working group is to identify new perspectives, methodologies and approaches from Management Science and Engineering that could help reduce death and suffering from cancer. There are many expectations in the new health care reform bill regarding what organizations should be doing to try and help improve healthcare value; however, most of these issues focus on primary care. Consequently, two questions have emerged: How do we bring specialty care into this debate? And what can research do to help guide these efforts? Although NCI does considerable work in cancer prevention and screening, these areas are already the focus of much research, as they fall under primary care. NCI would like to build its research portfolio to focus on the entire cancer continuum, targeting diagnosis to end of life. Management Science and Engineering is an exciting new approach that could be used as a tool to help build this program and provide information to achieve NCI’s objective (reduce death and suffering from cancer). This approach will be challenging though, since NCI’s current health services research is focused on individuals, e.g. the patient, the clinician, etc. The focus of research is rarely the health organization, an important element in cancer care. It will be important to answer, “What performance characteristics should health organizations have?” and “What tools are needed in order for organizations to achieve the desired performance objectives?” NCI is interested in learning how lessons from Management Science and Engineering can contribute to help build NCI’s research program to answer these questions.
Opening remarks by Karen Sepucha, Senior Scientist, Massachusetts General Hospital
Researchers need to rethink how delivery of cancer care is organized; however, this is a complex and complicated issue. Some key questions emerge such as, Where do we start? Where do we
focus our attention? How do we identify what really matters? The goal of this meeting is to identify areas of focus that will make a difference in the quality of cancer care. The Institute of Medicine (IOM) defines quality of care as, “The degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge.” It is important that healthcare providers are meeting patients’ needs and delivering effective, evidence-based care. To guide changes in quality of healthcare, the IOM proposed 6 aims and 10 rules for redesign. When examining these items, three themes emerged, which are the organizing principles of this workshop (Table 1). Care must be knowledge based, patient centered and systems minded. The IOM and the National Academy of Engineering (NAE) published a report on building a better delivery system, in which they identified promising applications, identified barriers and resources to implementation and identified areas of research. The key messages in this report affirm that Management Science and Engineering approaches have successfully transformed many industries and are already being implemented in healthcare delivery. However, there is still a lack of awareness from both engineers and healthcare providers regarding the potential of this science in healthcare and there is a limited track record for engineering in the service sector. Dr. Sepucha pointed out that conventional wisdom and readily available data (e.g. treatment rates) may not be the only or best way to approach the problem. Rather, it may require careful examination of data that really are linked to outcomes of quality that would result in more insights. The goal of this meeting is to identify key ideas from Management Science and Engineering in order to develop priorities and recommendations to improve cancer care delivery. These recommendations will be used to develop a strategy for building NCI’s research portfolio in this area. Questions of focus include: •
What are the high priority research questions?
What are the core capabilities needed to identify, adapt and implement evidence-based interventions, methods and innovations that improve delivery of cancer care?
What’s available now that can be adapted and examined in cancer care immediately? What is coming on the horizon?
Where are the gaps? Who and what is missing from the table? What needs to be developed?
What are the next steps that will help advance the work?
Session 1: Systems-Minded Cancer Care
Multidisciplinary Cancer Care Andy Salner, M.D. Director, Gray Cancer Center, Hartford Hospital
Dr. Salner discussed multidisciplinary teams and their influence on cancer care quality. What types of systems approaches are needed to facilitate both knowledge based and patient-centered care? These approaches should help accelerate the 3 “C’s” of Multidisciplinary care (MDC): coordination, collaboration and communication. Although cancer care has gradually moved towards specialization and away from generalists, many silos of single specialty and sequential care still exist with patient care arranged around physician schedules and location. However, there has been a recent movement towards multispecialty group practices, centered on collaboration and prospective patient management and coordination of care. MDC teams are engaging not only physicians, but also nurses, researchers, social workers and nutritionists, to name a few. The question remains, however, as how to align all providers involved, from the physician to the hospital or cancer center. Although the motivators for MDC should be patient centered, this is not always the case. Some are motivated to engage MDC in order to remain competitive with other local cancer centers. Physicians need to be community centered, meeting patients where they want to be treated. Primary care physicians should be involved as they are the source of new patient referrals. The infrastructure, staffing and communication technologies are also critical. A collaborative environment is crucial for fostering coordinated care and enhancing concomitant therapies. There are numerous reasons why healthcare providers are trending towards MDC; however there is no evidence to support that MDC is actually better. Research is greatly needed.
Relational Coordination as a Means of Measuring the Quality of Collaboration Jody Hoffer Gittell, Ph.D. Associate Professor, Brandeis University
She began by presenting evidence from a study of the flight departure process. This study found that relationships, not just the exchange of technical information, shape the communication
through which coordination occurs. The relationship dimensions that underlie effective coordination include shared goals, shared knowledge and mutual respect, which drive frequent, timely, accurate and problem solving communication. Problem-solving communication means that when problems arise, the focus is on “What went wrong?” and “How it can be fixed?” rather than on blaming and finger-pointing. High quality communication, in turn, reinforces shared knowledge, shared goals and mutual respect among care providers. This mutually reinforcing process is called relational coordination, defined as the coordination of work through relationships of shared goals, shared knowledge and mutual respect. Conversely, in poorly coordinated relationships, highly specific functional goals exist without regard to the “big picture.” There is a lack of knowledge of others’ tasks as well as status differences that tend to promote a lack of mutual respect. The result is communication that is infrequent, delayed, inaccurate, and focused on blaming rather than problem-solving. The same coordination challenges found in flight departures are also found in patient care. In fact, the most common quality problems reported by physicians are related to coordination. In the current system, each discipline defends its authority at the expense of the total system. Numerous studies in the healthcare setting have demonstrated the positive effects of relational coordination. Relational coordination enables healthcare organizations to increase quality and efficiency performance while enhancing working conditions for providers. For example, it has been shown to enhance care provider outcomes such as increased job satisfaction and reduced burnout or emotional exhaustion, while improving patient satisfaction, clinical outcomes, and reduced length of stay. It enables providers to connect in a meaningful way across functional and organizational boundaries. The question then becomes, “How do we build the organizational dynamics that support relational coordination and sustain it over time?” The basic organizational infrastructure needs to change in order to enhance coordination as well as sustain this change over time.
Challenges in the Continuity and Coordinating of Care Anne Miller, Ph.D. Assistant Professor, Vanderbilt University Medical Center
Dr. Miller presented a case study of a lung transplant patient whose care was not well coordinated. Although a plan was generated for the patient’s care, it was not well recorded or able to be tracked. The role of each team member was clear, but his or her interaction within the team was not clear. The patient’s care suffered as a result. One approach to improving coordination of care is to determine: Who are the team players? What are their roles? And how do they relate to one another? This data can be defined as the information environment, consisting of different categories of information such as diagnoses, expectations, goals, data and interventions. Each team member may occupy a different niche of the information environment, and the content focus of each niche and team member could be very different. Different disciplines may have different priorities. It is important that team members understand the roles of others and the potential implications for patients. Overt communication and team coordination must occur. This involves 5 steps: preparation (team members, schedule), planning (implications), direction (priorities, goals, plan), execution (time-line, next steps) and assessment (outcomes). Typically in the current environment, each of these steps is supported by a different information system. For example, diagnoses can be found in the electronic medical record and prescriptions can be found in the physician ordering system. Unfortunately, one system does not communicate with the other. This occurs among most information systems and creates information silos. The task of coordinating the information flow falls to team members, who must construct meaning from increasingly diverse and fragmented information sources. This may further increase the risk of communication breakdown and error. The followings questions need to be answered to address these communications issues: How should coordinated care be supported among team members who are separated by time and space? How do we track care goals? And do care goals really matter? Often, they are not the focus of care delivery. Who acts as the care coordinator director? Can information technology be designed to support distributed teams and care coordination?
Session 1 Discussion key issues and questions: •
What type of data is needed to support or evaluate the impact of MDC? Need more than clinical outcomes, such as disease free survival, also need patient empowerment measurements, patient satisfaction, and time to care
Need prospective model in the community setting showing that these outcomes are better with MDC. The majority of care (90%) is given in the community setting.
What is the goal of MDC? Is it that we are trying to get something like relational coordination through MDC? What is the ideal MDC team and how does it function (e.g. team meets with all patients and discusses all patients)? What is the practical or minimal MDC team (the team meets and discusses subset of patients, but only one providers sees the patient)? What are the tradeoffs with each?
Metric development is needed in order to develop data to evaluate MDC. Metrics should be incorporated into models. However, it is difficult to analyze systems and create models when focusing on the individual. Focus should be on collections of people. Personalized medicine cannot happen without population perspective.
Sustainability is important and it flows from systems that are based on the collective, not the individual.
How do we sustain change? Technology is useless without the social processes to implement it. Compliance will increase with technology but will eventually degrade unless socially integrated. Social integration must change before the technology can be successfully implemented. How can social integration be measured?
Providers need to learn about shared decision making.
Need systems to support safe practice. Care providers cannot be perfect 100% of the time given the complex nature of the healthcare system. Healthcare is a system of executable processes and support systems are needed to ensure patient safety in light of human error. There are core principles in engineering that can be applied to healthcare to address this issue.
How do we translate content into design? Care providers (e.g. nurses, doctors) are using the same information, but they are using it differently.
Applying systems to cancer care is challenging. In many cases, the patient will die and it is difficult to determine whether death is due to variations in care or progressive disease. Is it the patient or the system? Prognostic cancer outcomes exist for the general population, but there is no measure of outcomes for individual organizations. Data is weak and the assumption of failure is high. Expectations of success are more diffuse in oncology than any other branch of medicine. This can be improved with good science and data. 25
The only measure of outcome in cancer care is disease free survival, unlike other areas of medicine, which have defined predictors of outcome. We need to develop intermediary outcomes, such as measures of patient coordination and team collaboration. Is there a correlation between collaboration and disease free survival? It’s important to select team members that will collaborate and participate in shared decision-making. Patients are most concerned with coordination of care.
Session 2: Knowledge-Based Cancer Care
Informed Choice at Dartmouth-Hitchcock medical Center Dale Collins Vidal, M.D. Plastic Surgery, Dartmouth-Hitchcock Medical Center
Dr. Collins Vidal discussed the informed decision process, or shared decision-making, for patients at Dartmouth Medical Center. The goal is to inform and engage patients as partners in the decision making process so that they can make good decisions about their care. Health interventions can be categorized into 2 broad categories: effective care and preference-sensitive care. Effective care is evidence-based care that all with need should receive, such as antibiotics treatment, pap smears or flu shots. Preference-sensitive care involves treatment choices with multiple options; however, this type of care is not always evidence-based. The benefits or harms of this type of treatment are variable and dependent upon patient values. Unfortunately, it is not the patients’ values that drive care decisions in most cases, but rather the physicians. While informed consent is the standard process for informing patients, in most cases, it fails to help patients understand specific risks and benefits of treatment options. Often the choice between competing treatment options requires “preference-sensitive” decisions. In these situations, the treatments may not be supported by adequate evidence or they involve trade-offs that can variably affect a patient’s quality of life. Ideally the treatment choice would take into account an individual’s values and preferences regarding the potential outcomes leading to an informed choice. In contrast, “effective care” is supported by strong evidence and those decisions are less dependent on an individual’s personal values and preferences. In cases where effective care is indicated, a recommendation for treatment may be more appropriate, along with a discussion of
the potential benefits and harms with the decision maker. This approach is more in keeping with the traditional model of informed consent. An ideal system would take this process a step further and ensure that decision makers are adequately informed. A recent JAMA publication by H.M. Krumholz proposed the use of a standardized consent form that includes a description of the procedure, the potential benefits and risks, other available treatments, experience of your healthcare team and cost of treatment. This type of consent would not only facilitate informed choice for patients, but also help providers identify gaps in treatment options. Although preference-sensitive decisions are highly variable, informed choice could help reduce this variation. In fact, 30% of patients will change their choice based on information they received about their treatment. To improve the quality and efficiency of the informed choice process better patient decision aids and web tools are needed. Electronic questionnaires are used at Dartmouth to screen patient knowledge, patient preferences, demographics and psychosocial issues to generate an automatic provider report, which facilitates better decisions and more coordination of care. Shared decision making incorporates evidence-based medicine and requires both the patient and physician to contribute information and participate.
How to Design Effective Research Teams That Bridge Disciplines Steve Witz, Ph.D. Director, Regenstrief Center for Healthcare Engineering, Purdue University
Two types of perspectives can be described regarding multidisciplinary research teams: instrumental and conceptual. Fundamentally, the concern of instrumental-based teams is problem solving, whereas the conceptual concern is acknowledgement that a discipline is limited and can no longer explain the observed factors. Formation of instrumental-based teams is driven by an entity outside of the team. Conceptual teams are internally developed, e.g. “Is my observation no longer explainable by my discipline and can collaboration close this gap?” Multidisciplinary teams in cancer care delivery are more instrumentally driven. The 3 themes of care (i.e. knowledge-based, patient-centered and systems-minded) define the complexity of the solution and the complexity helps define the disciplines that should be involved. For the Regenstrief Center, the outcome is defined by the translation of research into clinical practice, not
knowledge, development or publications. The conceptual foundation in non-academic medical centers differs from academic centers. There are broad variations in perspective regarding research, methodology and risk. One of the biggest barriers, however, is the translation of research into practice, as each team member brings a different perspective. Therefore it is important to develop a dissemination plan that takes anticipated research findings and implements them into practice guidelines. Those that will be applying the research should be involved to verify the pertinence of the research. Teams may be interdisciplinary or multidisciplinary. Characteristics of effective multidisciplinary teams are based on members’ experiences with each other, communications and short-term intensive collaborative experiences. Studies at the Regenstrief Center show that the best multidisciplinary teams learn from other members’ disciplines and perform better with short, intense collaboration followed by individual research. Funding incentives have encouraged participation in weekly research team meetings and biostatisticians have greatly facilitated proper handling of data. Regenstrief has used a matrix management approach to disseminating research to translate it into practice. For this to occur on a national level the elements needed to achieve the desired objectives must be defined and interactions among researchers, practitioners and patients must occur.
Session 2 Discussion: •
Decision aids are always helpful, given the short interactions patients have with providers.
Patient navigators are very useful from a patient perspective, but should not be used to navigate around a “broken” system. Rather the system should be fixed. The navigator should not replace conversations among physicians - it should stimulate them.
Not every patient wants to be involved in the decision-making process. This must be preference-sensitive. Screening questionnaires could be used to assess how much information patients want.
Is MDC really better? Perhaps MDC is less efficient. We need evidence that this approach is better and more effective. How do we speed up translation? More peer-to-peer communication is needed.
If a model can be developed to show that cancer care can be delivered in a better way, which is knowledge based and patient centered, how could that model best be disseminated? It should occur through products that are accessible and affordable. Metrics are important. 28
The key to developing better care models is to engage practitioners in the design of research. Practice-based research networks could shed light on context, i.e. where will the research be implemented? Consensus may be reached more rapidly with this approach.
Perhaps changes in care should occur through data-driven experimentation. The idea is that it is better to do something, rather than nothing at all. If the process works, then continue. If it doesn’t work, try something else.
How do we bridge the gap between what we know and what we can do? Research findings should be accessible and linked to outcomes that the practitioner cares about.
It is important to define the change process. It involves changing the structures as well as relationships, and defining the sequence of events that works best.
Session 3: Patient -Centered Cancer Care
Meeting Patients’ Needs in the Community Peter Eisenberg, M.D. Medical Director, California Cancer Care
Dr. Eisenberg discussed the challenges for delivering patient centered care. Most patients have several expectations regarding their care: an explanation of their illness and its implications, results of the best-designed studies, assistance and support for their care decisions, compassion, and real patient advocacy. It is important to keep patients expectations in mind. Some patients do not want all the information. Treatment options should be discussed with risks and benefits, as well as the disease natural history (or trajectory) with and without treatment. A National Cancer Policy Board study concluded that for most people with cancer, there is a large disparity between the ideal and the reality of their experience with cancer care. Evidence shows that some individuals with cancer do not receive the treatment known to be effective for their condition. This may be a result of two issues. Patients in rural communities or with little money or health insurance coverage do not have adequate access to medical care. Conversely, patients with adequate access may be getting too much care that is not necessarily effective for their disease. He presented Quality Benchmarks that have been advocated by Joe Simone that include: care based on evidence from data or validated experience, care given in a professional, serious 29
learning environment, care that is accurately and fully recorded, and care given with sensitivity, efficiency and economy. So, who should be responsible for quality care? The government is not good at regulating the doctor-patient relationship and payers have little interest or rewards for quality care. Professional caregivers are best suited to take responsibility. However, the economics of chemotherapy administration often change the practice. For example, several years ago, many physicians treat patients with pamidronate, the generic form of Zometa, because of its higher profit for the practice, despite the fact it’s infusion time is nearly 2 hours longer. Incentives should not favor reimbursement over patient benefit. Patient centered care should be in the patient’s best interest, not the physician’s. Care should be offered in a thoughtful, unhurried, cogent manner. This implies that providers should have the capacity to spend extra time caring for those who require more hand-holding, a family conference, a literature search or e-mails or phone calls to consulting physicians. Patients, payers and politicians want care that is accessible and prompt, full disclosure of information and options, adherence to guidelines and evidence, patient choice, kindness and compassion, and emphasis on quality of life. However, there are several challenges to organizing cancer care to meet patient’s needs: perverse incentives, fragmented systems, poor uptake of new discoveries, and poor technological tools. Physicians should be rewarded (reimbursed) for the types of behaviors that are truly patient centered. Unfortunately, payers values procedures over cognitive skills. Systems are not interconnected as to provide quality coordinated care and technologies such as electronic medical records are poorly structured. When practice-changing information is published, tools to enable doctors to implement the changes are absent. Ultimately time and money are the biggest challenges.
Patient Centered Communication Neeraj Arora, Ph.D. Research Scientist/Program Director, Outcomes Research Branch, NCI
Cancer care focuses on survival as a primary endpoint; however there are many other important aspects, such as quality of care. There is a high failure rate in oncology – 40% of patients will die from their disease. This does not mean, however, that 60% of cases are successful. Survival, as
well as quality of life, is important. Patients have basic expectations for receiving cancer care. They want physicians to be knowledgeable and the care institution to provide the most effective care for their cancer. They want to know that clinicians will help in the decision making process and that they will receive benefit from their treatment and not harm. Patients also want to know that they will be treated equitably, i.e. they will receive the same care as any other patient. These factors should be the bare minimum for standard of care. Unfortunately, quality of care assessments has been largely focused on technical indicators, such as evidence-based interventions and comparative effectiveness studies. Patient centered measures have received little systematic attention despite the fact that patients have several needs in addition to survival. They expect informational, decision-making, instrumental (navigating the health care system), self-management (general health) and emotional support. Patient centered care should emphasize the whole patient. It should provide ongoing support to meet patients’ needs, both medical and psychological, and incorporate the patient’s perspective in care planning and delivery. It should also consider care within the context of patient relationships – not just the doctor-patient relationship, but also family members. It will be important to determine how to facilitate these relationships over the course of care.
NCI has been collecting data to measure delivery of patient centered care by performing a biennial survey, called HINTS, which evaluates national estimates of quality of care. In the most recent survey, they have incorporated measures of 6 functions that healthcare systems should support: exchanging information, making decisions, fostering healing relationships, enabling patient self-management, managing uncertainty and responding to emotions. The most recent survey demonstrated that approximately 15-30% of patients were not satisfied with their care in these 6 functions. If this data is translated on a national level, 24-50 million patients are not having their basic needs for patient-centeredness met by the healthcare system. Ironically, the national distribution of the cancer population is also 24-50 million people. It is important to have quality indicators of patient centeredness in addition to technical indicators of quality care. It is important to monitor and track over time whether providers are meeting the needs of patients. There are several challenges to doing this, as cancer care is a complex web of communications. Long-term longitudinal tracking of communications is needed; however, this is difficult to measure. The patient and the family have multiple interactions with multiple clinicians and these
interactions happen across multiple institutions. How can these communications be optimized? How can one model a complex system such as this? How can progress be measured? Quality and technical indicators are needed and better assessments of patient-centered care need to be developed. Technological tools, such as medical records, need to be integrated into clinical systems.
Patient-Centered Design of Delivery Systems Jim Conway, M.S. Lecturer, Harvard University School of Public Health
Mr. Conway provided several examples of patient stories to illustrate both the importance of, as well as the improvements in, patient-centered pediatric cancer care at Dana Farber Cancer Center. In many cases, managing care becomes paramount over the cancer itself. Hospitals tend to treat pieces, not the whole patient – there is no continuum of care. In the 1970’s, the Parent Advisory Council and a resource center were organized at Children’s Hospital, one of the first such centers in the country. This council provided parent engagement and power. From these experiences, 4 key concepts of patient and family centered care were developed: dignity and respect, information sharing, participation, and collaboration. In 1996, Dana Farber began a quest to become a national leader in quality, safety and patient- and family-centered care. They invited patients and families to participate in all decision-making bodies and organizational structures, including quality, safety and operating committees, leadership interviews, new employee orientations and resource centers. Many themes emerged from these experiences: the need for care continuum support, including second opinions, designing handoffs and advanced care planning; availability of resource centers for patient and family education and community outreach; there are no cookie cutter solutions – providers need to meet patients and families where they are. (Conway et al 200) So, how does one know the best place to receive care? There is currently no way to validate which institutions provide good care.
In order to achieve sustainable results in quality and safety, outcomes must happen at 4 levels: environmental, organizational, micro-system (process) and experience (patient and community). Although there is evidence for patient engagement in each of these levels, there are no
connecting threads. An unfragmented framework is needed. An initiative in Massachusetts was implemented in two low-income, working class communities to change consumer understanding of health initiatives. Approximately 94% of consumers understood that an informed patient receives better care; however, they did not know that they could ask questions. The campaign promoted three critical messages: carry a medications list, wash your hands and smart patients ask questions. After 18 months, understanding of these issues was better in these two communities than any other in Massachusetts. This study illustrates the power of patient engagement. The issue is no longer “if” patient centered care should occur, but rather “when” and “how”.
Session 3 Discussion: •
Institutions should be more transparent. Mortality and morbidity rates should be published. Errors should be disclosed to families. Transparency may also increase collaboration and competition.
How can effectiveness of transparency be determined? Most institutions currently use the same patient experience tool - Press Ganey. However, most of what is learned by this survey can be learned through patient advisory boards.
Everyone is measuring patient experience. What needs to be measured is care continuity. One of the biggest problems is patient discharge from cancer providers back to community providers.
Provider systems need to be developed to track where patients are going, how many times, etc. These systems should be linked to patient systems. When patients track their own chemotherapy, they begin to ask other questions. Involving patients in this way makes them a partner in their care decisions. When patients have an investment in their care, they become more responsible for improving outcomes.
Although patient navigators add further costs to care, they may increase patient engagement and decrease costs elsewhere in the system.
It’s not enough for patients to have good relationship with each doctor. Communication among physicians is one of biggest indicators for perceived patient care. A better organizational structure is needed to sustain this type of communication. 33
Impact counts, not p values.
What are research questions? The group suggested that one question be clarified before moving forward: Is the purpose of this working group to help forward a research agenda or an implementation agenda? In medicine, a greater distinction is made between these areas that is not made in other disciplines. There is not much distinction between the two from an engineering standpoint; however, it is important to identify the research questions, e.g. How do we advance knowledge to build more effective cancer care delivery systems? The most common need proposed by the group is the need for metrics. What is the performance index for cancer care? Implementation is often associated with an outcome, but the detailed components of the outcome are often missing. The implementation itself, as well as its associated outcomes, should be measured especially when dealing with systems metrics. One example is relational coordination. Compelling evidence for implementation changes is needed in order to move them into the community.
The group discussed the lack of comprehensive performance index for cancer care. Overall disease free survival is used for therapeutic interventions and there are a number of quality of life measures used as health status indicators; however, when it comes to patient centered care, everyone is using different systems and different metrics (or not using anything). There is no consensus around measures of patient centeredness. HCAHPS is one example of hospital quality improvements that is criteria, not satisfaction-based. Much of cancer care, however, is outpatient based and needs validated instruments. Although many outpatient measures were created before inpatient measures, their evolution has been slower. Picker was used for many years in the outpatient setting but was replaced by the CAHPS consortium; however, HCAHPS had the most merit in terms of quality improvement results. The ambulatory CAHPS has been used by CMS. Press Ganey is currently the most widely used instrument. Unfortunately, they are not part of the public domain like the CAHPS system. In terms of cancer care, both Picker and CAHPS instruments have important information, but do not capture the entire complexity of the patient
experience. AHRQ is currently developing a cancer version of the CAHPS instrument and working with NCI’s Outcome Research Branch to implement measures of patient-centered communications into the system. Some challenges lie in the level of depth of the survey, e.g. should questions focus on one particular doctor or all doctors, and how can this data be used for quality improvement purposes. On the other hand, the entire patient experience cannot be taken into account if only one hospital or center administers the survey. That particular center may not be interested in the patient’s experience outside itself. So far, the hospital version of CAHPS is limited to one center; however, cancer patients could be treated at many centers and clinics. Consequently, there is a great challenge in tailoring the instrument to the patient’s experience. This could also potentially threaten the anonymity of other aspects of the patient’s care. One suggestion was to tailor systems to the different stages of care. Dartmouth Hospital worked with Picker to tailor the survey to each stage of care, asking patients to skip sections that were not relevant to their care. In addition, more global questions were asked about patient doctor communications. These surveys, however, do not necessarily address quality of life or health or psychological status. On the other hand, adding more questions to surveys increases the burden to patients and decreases the response rate. A minimum data set needs to be developed that would be routinely collected.
QOPI is the Quality Oncology Practice Initiative sponsored by the American Society of Clinical Oncology (ASCO). Several hundred practices and several thousand doctors participate every 6 months by entering data into an ASCO website, which answers questions designed to measure quality of care. Each center can compare their score against the aggregate. Most of the measures are process measures not survival data, e.g. are all stage III colon cancer patients offered and receive adjuvant chemotherapy. It also measures aggressiveness of care, e.g. what percent of patients received chemotherapy in the last month of life, participation in hospice, days in hospice, etc. The measures are primarily outpatient and practice based. The website also addresses some measures of patient centeredness, e.g. patient education, personalized written treatment plans, etc. QOPI data is generated by the participating practices, not the patient. ASCO may also perform site visits to verify the data entered by participating doctors. QOPI data are an example of data that could potentially be used as intermediate outcome measures, i.e. the steps
along the way between intervention and the outcome. However, QOPI focuses on the medical oncology process and not integrate the entire cancer experience.
The purpose of this discussion is to identify outcome measures so that care delivery can be compared against measures and used to assess gaps. There is no set of outcome measures yet. The concern is that much time will be spent on developing outcome measures, which will eventually be ignored by practitioners due to CMS reimbursement rules. Various institutions must align to develop an agreed upon set of measures, which are non-proprietary. It may be difficult, however, to develop metrics when the outcomes are not established. Currently, there is no established method for testing whether interventions work or not. Even if metrics such as relational coordination could be measured, it may not be possible to link them to improved outcomes. There are many measures available, but a standardized set is needed that everyone can agree upon. NCCCP has organized a MDC research group to develop intermediate outcomes to study whether MDC care is efficacious. Two of the proposed metrics were time to definitive care and length of hospital stay. The idea is that formalized, coordinated MDC care results in a shorter time to care after diagnosis and a shorter hospital stay. The third metric is a tailored patient satisfaction survey. The goal is to ultimately to look at all domains, such as survival and quality of life and to have data that is accessible through the American College of Surgeons (ACS), the National Cancer Databank and rapid quality reporting systems.
A prospective study could be designed to look at coordinated care and patient centeredness: What outcomes would be used? What institutions would be included to involve both community and university based entities? Do patient navigators help? Do disease specific tumor boards make a difference? NCI is interested in studying patient centered processes, from patient centered communications to clinical and psychosocial endpoints, as there are few studies in this area. The idea of using intermediate measures as diagnostics is appealing. For example, relational coordination can be examined as a single measure for the whole overall process or it can be broken down to identify weak points in the process. Interventions can be made at these points, which serve as intermediate and diagnostic measures. It would allow redesign of the system to support and improve the intermediate outcomes. One important issue is the integration of community based and academic based cancer care. Perhaps the focus should be on developing
models for private care offices, which provide care for 85% of cancer patients. The challenge will be in developing a model to achieve these outcomes. There are few examples of good integration between community and academic centers. It is difficult to measure the impact of an organizational innovation unless it is studied within a patient population where clinical interventions are all the same, as there is much variation in cancer care.
Key Issues •
Metrics – What are accepted performance index measures? How can the size of implementations be measured? Progression free survival is one accepted measure but quality of life measures are needed.
Compelling evidence is needed for change to move into the community.
Everyone uses different measures – continuity is needed.
Most systems do not capture the whole patient experience. A minimum data set is needed.
What are outcome measures? Is MDC really better? There are many measures of patient experience but clinical measures, such as relational coordination, are also needed.
Clinical measures need to be standardized – they should include timeliness of care, length of stay and patient satisfaction surveys.
How can a model be built to achieve the desired outcomes? It is difficult to measure impact of an organization when there are so many confounding factors – treatment is highly variable.
Need to define relevant outcome measures, both long-term and intermediary.
What are the gaps? Who is missing from the table? Safety – there is a group among the comprehensive cancer centers called C4QI (Comprehensive Cancer Center Consortium for Quality Improvement), which works on safety. There is also a group emerging at Berkley with participation from UCSF – the high reliability organization network – that formerly focused on high reliability settings in industry, but is now moving into healthcare. The goal of the network is to utilize high reliability techniques to achieve patient safety. Currently, there are not many safety checks from provider to provider, i.e. from 37
pharmacist to nurse to patient. Quality indicators in oncology are behind many other medical specialties. ONS (Oncology Nursing Society) is working with ASCO to build some of these issues into practice-based standards. NCI needs to collaborate with other organizations that address safety issues, such as AHRQ. Safety and high reliability techniques also relate to coordination and collaboration. Studies have shown that collaborating is the key way to avoid errors from occurring. Some of the same systems that lead to high quality outcomes on a clinical level are also conducive to safety outcomes.
The strength of the discussion today relates to systems issues that can advance the quality of care that is delivered to patients. Many of these systems relate to coordination, collaboration and patient centeredness. The first step is to identify high priority areas where the biggest impact can be made. Then, experts in these areas should be identified and engaged to help facilitate research design efforts. Experts would include not only outcome experts, but also the institutions that will participate in the research. Researchers involved in developing survey systems such as Picker and CAHPS should also be included.
Key Issues •
Reliable and comprehensive network. Quality indicators in oncology are behind compare to other fields.
Systems – what are the high priority areas where biggest impact can be made? Coordination of care. Research.
Safety and IT – need better development of better tools, such as electronic medical records and better coordination among systems.
Needed collaboration among research efforts, e.g. NIH, AHRQ, ASCO, ONS, etc.
What is highest priority issue? What are you most excited about/want to pursue most?
Quality indicators are currently focused on clinical outcomes. Patient-reported process and outcome measures need to be brought to the same level. Data related to relational coordination
issues should be collected from both those who receive and those who deliver healthcare. A set of ideal metrics needs to be developed.
The incentives and desired behaviors from providers should be aligned. Also, tools are needed to make the current best knowledge accessible. For example, research publications that could impact quality of care do not quickly reach the practicing clinician. The research article does not append a set of standing orders or include educational pieces that would make the adoption of the finding easy.
The most difficult issue to change, but perhaps the most impactful is systems. The challenge is to find the experts in macro changes, such as what will be the impact of the trend toward centralization of cancer care (with these large oncology groups merging and will it be possible to be an individual practitioner in the future?), changes in the marketplace, changes brought about by healthcare reform, etc. How does one leverage these changes or take advantage of this environment to encourage the desired outcomes?
Most exciting is the possibility of developing metrics that assess outcomes and determining whether efforts of transparency could indicate quality. There are many established clinical outcomes, as well as many patient assessment tools regarding quality of life; however, intermediate metrics and assessment tools are lacking. Oncology care could be revolutionized if metrics could be developed and validated to assess systems, hospitals and providers and provide this information to the patient; however, the specifications must be defined to address evidence based care.
A better understanding of organizational practices that improve coordination on a system level is needed. What does this look like in the context of cancer care? The processes that lead to sustainable change also need to be identified. The risk is that the goal of this project becomes so large that it cannot be done. The focus should be on coordination and collaboration, and facilitators and barriers. There are different systems of oncology care. Variations that foster coordination and collaboration should be investigated at different organizations to assess whether
they are successful or not. The more specific we are in defining coordination and collaboration, the more successful we will be. There is an urgency to unbundle coordination.
Social media is an exciting, yet unexplored area for patient communication. Networks are needed for patients to share information with one another. Sites like Twitter or Facebook could facilitate positive information sharing. Currently in Massachusetts, a statewide model of public engagement is being built that engages hospitals, outpatient departments and communities.
Key Points What is highest priority issue? •
Patient outcomes and quality indicators
Desired behaviors and incentives should be aligned
Systems change – strong systems orientation
Metrics – need evidence to facilitate change
Improve coordination at the systems level and understand change process to develop sustainable change
Better coordination and collaboration
What are you most excited about/want to pursue most? •
Impact of relational coordination on quality of cancer care – design prospective survey to explore within multiple delivery systems
Opportunity to devise tools to facilitate physicians work and enhance patients’ experience, i.e. tools to make current knowledge accessible
Utilize transparency to assess systems
Devise methods and metrics to assess outcome and measure improvements
Explore social media to get information to patients or retrieve information from patients
Build a model of public engagement and cancer care
Next steps The challenge is in moving this project forward, gaining support and taking the next steps. One or more intermediate steps are needed before considering a specific research project. The ideas 40
from this meeting should be broadened to get more input from a broader group of experts. Experts should include knowledge transfer, dissemination, social media, and communication, among others. There is a lot of activity that needs to be shared to make good decisions that is not yet in journals. If NCI could fund a research project, could an RFP be designed? More detail, specificity and evidence than discussed at today’s meeting would be needed. In addition a constituency must be built to facilitate and support the process. To build a strong constituency, coordination among organizations such as ASCO, QCCC, NCCN and ACS needs to occur. One venue for presenting the ideas discussed at today’s meeting is the Cancer Quality Care Committee, which consists of 12 federal agencies. There are several other venues as well that could be used as discussion points to facilitate today’s agenda.
Key points •
Prioritize key areas – identify specifics
Build constituency for changing care
Indentify and engage more experts such as dissemination and translational experts – need a broader context
Coordinate with other organizations, such as ASCO, QCCC, NCCN, ACS
Appendix B. Key Informant Interviews We interviewed four cancer specialists, several of whom also have administrative responsibilities in their organizations, about the key challenges and opportunities they see to improve the quality of the delivery of cancer care. The excerpts here have been edited from interview notes and serve to highlight some of the similarities and differences in opinion along some of the topics that were raised in the strategy paper. For example, in our interviews, we heard very different opinions about multidisciplinary, teambased approaches care, which is one of the more significant organizational changes that has occurred (in some places more than others) in cancer care. Folks from administration have given us a clear message that we have to have these multidisciplinary clinics. This is more for marketing not patient care. Our team already works well together and it is not completely clear that the multidisciplinary clinic is better for the patient (very long day) or the provider (lots of wasted time). Doctor 1 One of the best things that has happened is embracing multidisciplinary care, the way I work with colleagues and how we collaborate on developing care plans – that really fosters sense of team, really helps our patients feel more supported. Doctor 3 It was interesting that both doctors who reported that multidisciplinary care was positive also talked about the importance of recruitment in creating the team: Sometimes it’s hard but we attempt to recruit people with like-minded approaches, commitment, and work ethic. Doctor 3 The work life balance is a huge issue and we have made a clear commitment to that in our practice, we don’t make as much money as possible but we each take eight weeks of vacation each year. Obviously that means we are always covering for each other and we have worked hard to share information about patients. Doctor 4
Conversely, not paying enough attention to fostering a true sense of team work was seen as problematic:
Cross coverage is best way to ensure that we learn from others, but this practice has virtually stopped. Our group tries to minimize cross coverage because you can’t get your billing if you are not there supervising the treatment, and of course no one ever shares their worst cases. Doctor 2 There was a variety of opinions about the usefulness of guidelines in routine practice. If you need to consult the guidelines you probably shouldn’t be taking care of breast cancer. I don’t use them ever in day to day practice. I do, however, regularly track our groups’ adherence to guidelines and we examine all the discordant cases to make sure we are appropriately delivering care. Doctor 1 I look at most recent updates almost continuously, every day. The guidelines cover an enormous amount well, except end-of-life situations, and the NCCN guidelines are an unbelievably valuable accomplishment. Doctor 2
These changes are not just about evidence, but they are about mindset, and fundamentally changing the way clinical decisions are made. While there are many impediments to providing high quality care, one of the primary issues is changing the mindset of the physician. Instead of asking, “What regimen can I sell this person?” one should be asking, “Who is this person?” “What do they want to know?” “How can I tell them in the best way so that they can make decisions for themselves?” and “What does the science tell us?” Doctor 4
Everyone had experience implementing EMRs and other health information technology systems. No one had an extremely positive reaction, though some were better than others. Implementation is long, expensive and poorly thought out. There is this rigid adherence to the fact that we have to do everything through the electronic system, even if it would make sense in other ways. It’s like having to take your car into the supermarket and shop through the window, instead of parking it and then walking around the store. Paper is just better for some things. Our organization appears to have adopted the goal of a paperless office, but that’s getting in the way of high quality patient care. Doctor 2 The system has provided many advantages, such as immediate access to patient charts no matter what the location, the ability for multiple people to view a chart at one time, and fewer charts being lost. However, the EMR system does not increase efficiency in
charting, does not prevent mistakes, and does not decrease the generation of paper documents. Doctor 3
Two of the doctors specifically mentioned email as an essential and important change that has markedly impacted their ability to delivery high quality care. For the most part, they were talking about using email not to interact with patients, but with other providers: Probably one of the best things to happen in the last 10 years has been email to experts (cheap second opinions). Doctor 2 Some of the biggest barriers they see to being able to deliver patient-centered care include lack of systems to help with coordination as well as challenges due to the financial system: Lack of care management systems that allow me to backstop whether people are missing appropriate care. I have no easy way to tell whether things are being followed through (e.g. if I refer someone for medical oncology and they don’t show up, I’ll never know). My workaround is that I routinely see patients at 4 weeks (after surgery) for the sole purpose of making sure that visit is not missed— though I realize that’s probably not the be best use of my time or health care resources. Doctor 1 The biggest issue I see is reimbursement. Medicare does not reimburse fairly. All the incentives are to be an imaging and chemotherapy delivery center not a center that takes care of patients. Doctor 4 There’s a population that cannot pay for the best, and at $100k a treatment there’s a big tension between caring for an individual and a population. How do we resolve these issues before the system crashes? Doctor 3
Appendix C. Additional background materials for working group The following background material was sent to all working group participants and was edited from a report “Understanding and Improving the Quality of Cancer Care Delivery: Insights from a systems model” that was produced by Karen Sepucha, PhD, for the NCI, under task order #263-MQ-516617.
In preparing for this meeting, we have organized the discussion around three key themes that are critical to high quality cancer care, namely that care is knowledge based (driven by science), patient centered (responsive to needs of patients), and systems minded (supported by infrastructure and resources to ensure this happens in timely, efficient, safe and equitable way). We briefly review the derivation of the themes and then expand on each in relation to cancer care.
THREE KEY THEMES Multiple IOM reports have generated broad awareness of the urgency to make changes in order to achieve measurable improvements in the quality of health care generally, and cancer care specifically. Quality of care is defined as the “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge (IOM 1990). To guide the changes, the IOM laid out six aims for quality care—that care is Safe (avoid injuries to patients from the care that is intended to help them), Timely (reduce waits and sometimes harmful delays for both those who receive and those who give care), Effective (provide services based on scientific knowledge to all who could benefit and refraining from providing services to those not likely to benefit), Efficient (avoid waste, including waste of equipment, supplies, ideas, and energy), Equitable (provide care that does not vary in quality because of personal characteristics such as gender, ethnicity, geographic location, and socioeconomic status) and Patient-centered (provide care that is respectful of and responsive to individual patient preferences, needs, and values and ensuring that patient values guide all clinical decisions). The report also laid out ten rules for redesign of the health care 45
system (see Table 2). The importance of cooperation among the clinical team, although it was not included as one of the six aims or the ten rules, was also emphasized as a cross cutting theme.
Table 2: Three Common Themes in the Ten Rules for Redesign and the Six Aims 10 Rules for Redesign
1. Evidence Based Decision Making
6 Aims for Quality 1. Effective
2. Cooperation among Clinicians 3. Shared Knowledge and Free 2. Patient-Centered
Flow of Information 4. Customization Based on Patients’
Needs and Values 5. Anticipation of Needs
6. The Patient as Source of Control 7. Continuous Healing
Relationships 8. Safety as a System Property
9. The Need for Transparency
10. Continuous Decrease in Waste
These aims and rules are not independent and if we consider how they overlap, there is an opportunity to highlight common themes. Table 2 presents the three core themes that capture each of the 10 rules and 6 aims—namely, that care is knowledge-based; patient-centered and systems-minded. These three themes provide a streamlined framework for thinking about quality that links directly to the proposed definition of quality of care. Care that is knowledge-based is “consistent with current professional knowledge” and will make use of the evidence to avoid overuse and underuse. Care that is patient-centered will “increased the likelihood of desired outcomes” by meeting the needs and wants of all patients, regardless of age, gender, or geography. Care that is systems-minded will have the appropriate infrastructure (e.g. information 46
technology, regulatory, financial, SOPs) in place to deliver high quality care and the systems in place to capture the outcomes for individuals and populations to facilitate continuous improvements in care.
These three themes have become the organizing principles for the workshop. We have designed three sessions that each focus on one theme. The following three sections provide some additional details on each area.
KNOWLEDGE-BASED CARE Ackoff (1989) describes the “knowledge hierarchy,” a progression from data to information to knowledge to wisdom that reflects increased understanding at each step. Data and information may have a fairly short useable lifespan, while the knowledge and wisdom are more lasting and allow for productive reasoning on prospective problems or decision situations. If examine the “knowledge hierarchy” in cancer care what would we find? How are the data collected, shared and analyzed? How is the resulting information collected, shared and analyzed for patterns? How is the resulting knowledge collected, shared and analyzed for principles? How is that wisdom collected and shared? How does “conventional wisdom” match up with evidence-based approaches to decision-making? Who in the cancer care system is responsible for knowledge management? What should be shared with patients (e.g. data, information, knowledge, wisdom) and how? Do patients have knowledge and wisdom that they should share with providers?
To answer these questions will shed light on the current system of knowledge management in cancer care. Data are collected and stored in progress notes, lab reports, imaging studies, tissue banks, and billing records – much of which finds its way into medical records. Patients receiving care from multiple specialists at multiple sites will have multiple records that are not easily accessible or shared. Many studies have documented missing or inadequate data or information in medical records, even electronic records and in cancer registries (Peabody et al. 2004; Wagner and Hogan 1996; Malin et al. 2002). Thus, there are many gaps in the collection of data during routine care clinical across the cancer care continuum.
The clinical literature produces information, knowledge and arguably, leads to wisdom. Consensus conferences and guideline developers also seek to influence wisdom through summarizing the literature and making conclusions about the appropriate actions in certain circumstances. Clinical trials, though valuable, represents the experiences of a minority of patients, about 5% of adult cancer patients enter clinical trials. (Notably, one of the most common reason for patients to not enter a trial is not being aware of, or having the knowledge that a trial was an option for them (Harris Interactive 2001 and Mills EJ et al. 2006)). Despite all of this information there is still a fairly primitive understanding of what should be measured and which factors really influence quality. An AHRQ report on the status of quality measures for breast cancer care found that of the 143 quality indicators used in studies only 12 (all dealing quality of life) had any evidence of effectiveness or validation (Schacter et al. 2004). There needs to be more work done to even identify which observations or data are most important to collect.
In cancer care, tumor registries such as SEER and the National Cancer Database (NCDB) of the American College of Surgeons, play an important role by collecting a limited set of data on tumor, treatments and outcomes for a large, more representative portion of cancer patients. Cancer registries support the analysis of patterns of care and provide important information on national trends in incidence and mortality. Epidemiologic studies and observational studies also capture the experience of a larger portion of patients, but with more detail than the registries. Several large outcomes research projects, such as, PCOS (the Prostate Cancer Outcomes Study), Capsure (CAncer of the Prostate Strategic Urologic Research Endeavor) and CANCORS (the Cancer Care Outcomes Research and Surveillance Consortium) provide more detailed longitudinal data on the experience of prostate, lung and colorectal cancer patients (see for example, Cooperberg et al 2004). The American Society of Clinical Oncology (ASCO) launched the National Initiative on Cancer Care Quality (NICCQ) in which researchers examined outcomes of a cohort of 1800 breast and colorectal cancer patients. To the surprise of many, the NICCQ results indicated that the quality of cancer was fairly high. The majority of patients, 86% of breast and 78% of colorectal cancer, received guideline appropriate care (Malin et al. 2006).
These more detailed outcomes studies provide a snapshot of the quality of care for only 25,000 patients and they are costly. The limited data captured in a more structured way on a larger portion of cancer patients through registries is too limited to answer many pertinent questions, and gathering the detailed data through large observational studies is expensive. Obviously, it would be best if there was a way to facilitate access to the data that are collected on patients during routine care, but that would require significant investments in infrastructure for information technology. Until then, the best data in cancer care will continue to be financial/billing data not clinical care (Esserman 2006).
Some cancer centers have made efforts to come together to try to overcome some of these limitations. Notably, the National Comprehensive Cancer Network (NCCN) has assembled a group of 20 leading cancer centers across the country to collect data on patients and share the learning between their members. They have also influenced the larger system through producing extensive clinical practice guidelines. A recent CMS demonstration project has actually tied reimbursement to the providers’ ability to document whether treatment was consistent with the NCCN guidelines. In order to accomplish this, the NCCN has set up a parallel data collection system at each participating institution, (data managers abstract data and enter it into research database) because the quality of data produced during routine care was not sufficient to use for their analyses (Weeks J, 2005).
A group of 10 large integrated health systems have also come together forming the Cancer Research Network (CRN) (Wagner et al 2005). The group is funded through NCI to answer important questions such as the quality of community based care, long term outcomes, and costs associated with cancer care and improvement programs. Projects span the continuum of cancer control from prevention to cancer care treatment and end of life care. As we mentioned earlier, many of these integrated delivery systems had more sophisticated information systems in place, however, the CRN had to overcome large organizational barriers to sharing data. Understandably, members were concerned about the sensitive nature of the data on quality and
costs, and instead of creating a centralized database, they invested in a developing standards and dictionaries to enable the comparison of data across the sites (Wagner et al. 2005).
The widespread recognition of the value of combining data across sites in our fragmented delivery system, has motivated the development of these virtual systems, like NCCN and CRN, from our fragmented delivery system. Both CRN and NCCN have had to overcome major obstacles both technical and organizational to sharing data and generating new knowledge. This raises some questions about the current practices and lessons to guide future collaborations. What are the major obstacles to creating these virtual networks to foster learning across patients? What are the incentives for providers, administrators and payers to engage in such networks? What kind of infrastructure is needed and who is responsible for it? The significant amount of rework that NCCN has had to do for its database, raises some other questions such as, if the data generated for routine care are not good enough for research are the data really good enough for patient care? What kind of efficiencies would we achieve if we focused on how to get it right the first time? How might a medical record with high quality structured data revolutionize epidemiology and observational studies?
With a few simple queries, a store manager at Walmart can tell you exactly how many red shirts were sold in the last week at what prices, and how that compared to the local, regional and national averages. The majority of oncologists cannot tell you how many patients they saw last week, let alone how many had a reduction in dose of chemotherapy, how many were bothered by severe fatigue, or how their outcomes compare with local, regional or national averages. Although the data are potentially available for assembling this information, it requires a significant amount of work and is not accessible by patients or providers to guide decisions at the point of care. Despite the fact that cancer care uses some of the most sophisticated tools and technology in the care of patients, the tools and technology it uses to manage the knowledge base and guide clinical decisions are antiquated and inadequate.
Most models of cancer care track the natural history of the disease, starting with prevention then moving through detection, diagnosis, treatment, survivorship and end-of-life. Although this may be a convenient mental model for researchers and clinicians, this is not how patients usually experience cancer, and delivering care that is truly patient centered may require a different perspective.
The diagnosis of cancer is an interruption in patients’ lives—not the beginning of patients’ lives—and it disrupts plans, family, career, friends, and hobbies. The tests, visits and treatments need to be incorporated into already busy lives, requiring that priorities be shifted and sacrifices made. More often than not, people have been through a “c-word” experience before, perhaps as a child, friend or spouse of a cancer patient. They also bring their religious views, or cultural views on health, medicine and cancer with them. For example in the Navajo culture, it is taboo to talk about cancer because to say the word means that you are inviting it in (Robinson et al. 2005). The connections in patients’ minds, the experiences both good and bad will color their approach to the disease—regardless of whether the medical situations are comparable. In a ‘diseasecentered’ approach to care, this type of information and experience is often ignored or controlled for so that decisions can be based on the medical facts and clinical situation.
As the perspective in cancer care shifts toward a more patient-centered approach, this will require rethinking of roles and relationships. It does not, as some suggest, mean that providers are no longer needed or necessary in the decision-making. Rather, it recognizes the fundamentally joint nature of action—patients cannot effectively diagnose or treat their cancer alone—they need the doctors, nurses and the care team. Likewise, the care team cannot diagnose and treat the cancer alone—they need patients. The recognition of the interdependencies between providers and patients raises some important questions for understanding and improving the delivery of patient-centered care. In particular, what knowledge and wisdom do patients have that should be used to guide decisions? How can we best engage patients in decisions and in their care? What should be shared with patients (e.g. data, information, knowledge, wisdom), when and how? How do we elicit patient’s needs and wants? Do patients’ needs and wants change over time or with increased knowledge, and if so how? What should happen if an 51
individual patient’s preferences for treatment do not match with conventional wisdom or with a public health perspective (e.g. can someone make a good decision to smoke, or to not follow the “standard of care” and decide not to be screened regularly for colon or breast cancer?) What are the competencies that health professionals need in order to deliver patient-centered care?
Patients need to take on some responsibility for ensuring that care is patient-centered. Patients are best suited to assess their needs and wants and to determine the impact of treatments on their life and to determine whether it is worth it. The need resources and support to do this well, but patients and families need to play a big role. This view is in contrast to a more marginalized, helpless perspective of patients that has dominated much of the literature, and it tracks with two growing trends in health care. First, the expanding role and responsibility of patients in chronic disease management and the association of increased self-management and improved outcomes (Lorig and Holman 2003). This movement has been described as a disruptive innovation, reflecting the considerable upheaval and changes involved as patients and other non-physicians play an increasing role in the delivery of care (Christenson et al 2000). As cancer care becomes more chronic in nature, this model will be important to learn from.
Second, the recent democratization of information has transformed the ability of patients to become informed and involved in their care for both acute and chronic illnesses. It was not long ago that only physicians had access to medical journals and the latest advances. The Internet has succeeded in making medical evidence and information available to a much wider audience. In addition to finding information from clinical sources, cancer patients also use the Internet in large numbers to join communities of other patients that share information, knowledge and wisdom about treatments and the disease. This type of information on the subjective experiences (e.g. what is it really like to lose all of your hair? How scary is radiation therapy?) and how to manage the many issues in the rest of life (e.g. dealing with emotional concerns, how and when to tell your boss about a diagnosis, how or when to tell a boyfriend that you are a cancer survivor) is not routinely captured (and not really considered “evidence” or “medical knowledge” yet the answers are critical for patients to understand their disease, make decisions and get back to a life worth living. 52
These trends of increasing patient responsibility are being mirrored by health plans in placing more of the financial responsibility in the hands of consumer. Increasing responsibility without adequate support can lead to problems, not the least of which are increased anxiety and information overload. Providers and patients need to adapt to the changing roles and responsibilities. Looking ahead, we need to invest in improved technology and applications that will facilitate patient involvement and more patient-centered care.
SYSTEMS-MINDED CARE There is a considerable amount of organization and infrastructure needed to deliver knowledgebased and patient-centered care. This reflects the content of the third theme, systems-minded care. In order to make sustainable improvements to cancer care we need to understand the different forces that enable and constrain the actions of patients and providers in the delivery of care. A systems approach differs from more traditional methods of analysis that promote a reductionist approach to understanding and optimizing performance. A reductionist approach will break apart a system into manageable sub-systems and then understand, control and optimize the performance of those subsystems. However, this approach only generates an optimal solution for the entire system when the subsystems are independent. Otherwise, what happens is the solution that may work locally, produces negative consequences in other parts of the system. And, attempts to combine the pieces to see the big picture is often futile—as some have compared it to trying to reassemble the fragments of a broken mirror in order to see a true reflection (Senge 1990).
In the United States, the fragmentation of the health care system does not make it easy to take a systems approach. This is mainly because the benefits and harms of a new intervention or policy often do not accrue in the same way to the same stakeholders. As mentioned earlier, Leatherman and colleagues (2005) found that the financial benefit of quality improvement initiatives often did not accrue to the investor, and almost never benefited the provider even thought they often had to make the most changes to implement. Even though it might be the “right” thing to do, the system did not provide the right incentives to make it happen. Payers and policymakers have 53
called for realigning incentives across the medical system and have started to pilot projects to pay for performance with the goal of improving quality. However, given the lack of accessible data about the quality of routine cancer care, a first step that might be more productive is to institute incentives to reward the collection of good data and to build the required infrastructure for data collection (Esserman 2006).
The places within the US health care system that have made some of the most impressive strides in developing infrastructure and systems to support care are those that are more integrated, like Kaiser and the VA hospital system. Over the years, multiple studies have documented increased satisfaction of providers in these closed systems, compared to those in open systems that accept patients from multiple health plans (Murray et al 2001 and Chehab et al 2001). However, these plans have also suffered from perceptions—real and imagined—of the dangers, problems and inefficiencies of closed, managed care systems. Although it is easier to recognize and measure the impact of interventions within these integrated systems it is still difficult to control the delivery of care.
Because of (or in spite of) the fragmentation of the medical system there are many examples of cancer researchers and others are coming together in informal or virtual systems. Many of these focus on research (e.g. the cooperative trials groups, NCCN, CRN), although slowly, cooperatives are being created that focused on implementation and delivery. One example in cancer is the American Society of Clinical Oncology’s Quality Oncology Practice Initiative (QOPI) program whose goal is to change the culture of oncology practices to one of continuous quality improvement. The idea is to enable providers to improve care by raising awareness of their own performance and that of their peers, and sharing best practices and tools for improvement. The pilot engaged 23 sites in collection of practice data on multiple quality indicators and the results showed wide variability on several indicators. (Neuss et al 2005, McNiff 2006). These data were then used as motivation to implement changes to improve quality. The leaders of QOPI have created incentives for practices to join, for example, working with the American Board of Internal Medicine so that participation in QOPI meets the requirements for compliance for continuing medical education. (Simone 2005) 54
Even though many may argue it is not a system (when asked, for example, what he thought of the health care system, Hank McKinnell, CEO of Pfizer and author of A Call to Action: Taking Back Healthcare for Future Generations, responded “I think it would be a good idea.”), the health care “system” in the United States can and should be explored and examined. What are the infrastructure or organizational needs that can foster more collaborations and networks (like CRN or QOPI)? How do we analyze and optimize the performance of large, complex systems? In order to answer these questions, health services researchers will need to expand and enrich their tools, methods and areas of inquiry. (Berwick, 2005) When trying to understand and improve systems, the classical methods of evaluative sciences, with the focus on randomized controlled trials and the advanced statistical analyses of epidemiology, all carefully designed to minimize confounding and bias, may not be as appropriate. What’s needed are methods that can explore multiple hypotheses (that are not clearly defined), each with multiple alternatives that may change over time, and that are responsive to a variety of local resources and barriers. A potential middle option is use of preference trials, or comprehensive cohort trials that combine randomized controlled trials with observational cohorts. (See for example McPherson et al 1997.) Patients with strong preferences for one treatment or another are allowed to choose and are followed in the observational cohort, while those who are indifferent are randomized. In this way, these trials gather information about whether and how patients’ preferences influence outcomes by comparing outcomes of groups who chose and those who were randomized. (See for example SPORT trial of treatment of back pain Birkmeyer et al 2002).
Peter Senge has studied and written extensively about systems thinking and organizational learning. He cautions that people tend to blame “the system” for their behavior (especially their inability to meet certain targets or goals), but with a few exceptions, the system does not make people do anything. The vast majority of what’s referred to as "the system" is actually habits and taken-for-granted assumptions of the actors within an organization—the local beliefs and practices. Thus, changing the “system” also requires significant attention to organizational change management, not just to the infrastructure or resources (as was evidenced in reports of the creation of NCCN and CRN). Individual patients and providers do not have the authority or
resources to make these systems-level changes. Although everyone in the system needs to participate in order to effect change, the responsibility for ensuring that care is systems-minded rests with the leadership and management – the policymakers, purchasers, and directors of the cancer centers.
Appendix D: Selected initiatives using engineering across NIH
There is increasing interest in examining how engineering can improve cancer care. A few initiatives have been launched across government agencies to explore these opportunities. Recently, AHRQ and NSF published a report, “Industrial and Systems Engineering and Health Care: Critical Areas for Research” (report #). As the title suggests, it focused on identifying areas where industrial and systems engineering (ISE) might help address challenges in health care delivery, with a specific emphasis on information technology (IT). Given the similarity with what we were attempting to accomplish, we summarize their findings and then discuss applicability and relevance to our goal of identifying opportunities for management science and engineering to improve the delivery of cancer care.
As background work, they reviewed 13 government reports dealing with health care and engineering. Then they invited experts to a two day meeting in order to accomplish three goals, to articulate a vision for the health care system, to identify barriers and facilitators to using ISE to achieve this vision, and then to identify key research areas. The invited experts focused on five ISE areas: 1. information technology, finance and quantitative decision making 2. systems change, analysis and implementation theory 3. materials management and production processes 4. Human factors and sociotechnical systems 5. Quality engineering They also examined cross cutting health care challenges:
1. Managing acute illness and disease 2. Creating effective models of health promotion and disease prevention 3. Insuring chronic disease management 4. Enhancing end-of-life experience 5. Facilitating public health 6. Accelerating discover As is evidence by the breadth of these topics and disciplines, the charge was very broad. Although cancer care fits into many of the health care challenges, it was not a focus on the session. In addition, it was not good enough to achieve improvements in just one area or aspect rather it had to be complete breakthrough change. Given the size and complexity and diversity of the health care system, this was a very daunting task.
Vision for a Health Care System: The vision for the health care system was one that was new, patient-centered and engineered. There was an extensive list of details, including that the system be integrated, ubiquitous, distributed, responsive, expansive, flexible and resilient. In addition, they described delivery of care as personalized, with secure information flow, transparent and open access. They advocated for a system that is “information optimized” and that runs smoothly, efficiently and safely. When compared with definitions put forth by IOM (six aims and ten rules of redesign), these had more of an engineering perspective. However, the vision did not distinguish between “means” (how to get there) and the “ends” (where do they want to go). If they had done more work to prioritize the goals (e.g. why is it important to have a transparent system or open access to data – what will that help us achieve?), it might not look much different than the IOM six aims.
Barriers and Facilitators to ISE: They identified nine barriers and four facilitators to using ISE to improve health care. Barriers included lack of current use of ISE tools and inadequate knowledge of ISE, and insufficient IT infrastructure. The lack of infrastructure to share knowledge, fund research and promote use of tools was also identified. The facilitators included things such as increasing recognition of ISE potential in health care and progress in recognition of need for new ISE tools to deal with health care problems. None of these barriers were very different from 57
those identified in the earlier NAE report (Engineering Health Care Delivery) and are likely the same as for any other engineering discipline (including management science and engineering). One issues that was new (a facilitator) was mention of the favorable political environment that may enable more change in health care.
Critical Research Areas: The report started by noting a general lack of clarity on what exactly constituted ISE, which posed a challenge to developing research agenda. ISE is very multidisciplinary by nature and routinely draws from mathematics, psychology, organizational behavior as well as other disciplines. The areas of study and potential solutions offered during the meeting ranged from technology and modeling to reforming tax policies or incentive structures, and increasing regulation. The former were more “traditional” engineering methods and the latter were often considered under the purview of other disciplines (e.g. economics, public policy). The lack of solid boundaries in engineering is a common problem, and shared by MSE (in fact similar issue came up in our meeting when asked to define MSE and really it is a catchall term for a wide range of techniques and methods). A further challenge was that participants did not agree with the background documents that argued that in order to achieve breakthrough change that simply transfer of existing knowledge is not enough, rather they identified several opportunities for current ISE knowledge could be applied.
Despite these challenges, the report documents an extensive research agenda (more than twenty pages of research agenda items). The broader research agenda was grouped into three main topics: (1) stimulate innovations in ISE methods to better address challenges in health care (2) accelerate knowledge transfer of available methods to solve current health care problems (3) integrate meta-knowledge lessons gleaned from research projects. The vast majority of the report focused on the first topic which was further divided into three areas, issues involving system monitoring (or ways to gather data about the health care system and share that with relevant stakeholder), system modeling (or ways to improve understanding of the system and how the pieces interact), and finally system modification (or ways to change and improve to the system).
The issues ranged from identification of best practices for dissemination and adoption of ISE knowledge to models that can be built from incomplete, inaccurate and unreliable data to frameworks that explore the integration of many care sources in the production and delivery of care services. They ranged from very generic (e.g. efficient methods for integrating large amounts of data from disparate sources) to fairly specific and tailored for health care (e.g. new automatic data collection technology to capture patient level data and environment such as sun exposure or food intake).
Finally the report presents an action agenda to be pursued by relevant funding agencies to advance the research agenda. The identified items in five areas that are needed in order to move ahead with the vision (1) need for collaboration to be established among Federal agencies (NIH, NSF, AHRQ and the VA) a list of 24 organizations (2) education and training for engineers and health care administrators and providers; (3) need for new sources of funding; (4) need for programs to support dissemination of findings; and (5) administration to support funding and speeding proposals through grant cycle.
In general, the background materials strongly suggested that ISE did not have knowledge currently available to enable wholesale health system change. Although there have been successes in individual companies there has not been evidence of methods and technologies that have led to breakthroughs in entire industries that could be applied to health care. At the end of the workshop, the conclusion was that ISE is not able, in its current state, to provide breakthrough change in the health care system. There needs to be more research and much more advances in methodology and tools. An extensive research agenda, with an extensive action agenda laid out an ambitious plan to generate new knowledge that may lead to breakthrough change.
The NIH Commons Fund Health Economics program released a call for applications in August 2010 focusing on Science of Structure, Organization and Practice Design in the Efficient Delivery of Effective Healthcare (R21). The motivation for the RFA was the Patient
Protection and Affordable Care Act 2010 which dramatically expanded health care coverage and provided for the evaluation of a range of approaches to managing growth in health care costs. The research call had an explicit focus on health economics (as opposed to engineering) and an explicit cost focus (as opposed to patient-centered care or high quality care), for example, one possible type of project was to indentify structural and organizational features associated with high and low costs providers. It is not likely that the call will result in significant insights for cancer care delivery (as most of the focus was on primary care, medical home and accountable care organizations), however, it is an example of cross cutting research that focuses on the delivery of care.
References Accelerating Successes Against Cancer: Recommendations from the NCI-designated Cancer Center Directors. US Department of Health and Human Services, National Institutes of Health, 2006. Ackoff R, "From Data to Wisdom," Journal of Applied Systems Analysis 16 (1989): 3-9. Berwick, D. The John Eisenberg Lecture: Health Services Research as a Citizen in Improvement. Health Services Research 2005 April: 4(2):317-36. Birkmeyer NJ, Weinstein JN, Tosteson AN, Tosteson TD, Skinner JS, Lurie JD, Deyo R, Wennberg JE. Design of the Spine Patient outcomes Research Trial (SPORT). Spine. 2002 Jun 15;27(12):1361-72. Bohmer R. Fixing Health Care on the Front Lines. Harvard Business Review. 2010 Apr;88(4):62-9. Chehab EL, Panicker N, Alper PR, Baker LC, Wilson SR, Raffin TA. The impact of practice setting on physician perceptions of the quality of practice and patient care in the managed care era. Arch Intern Med. 2001 Jan 22;161(2):202-11. Christensen C, Bohmer R, and Kenagy J. Will Disruptive Innovations Cure Health Care? Harvard Business Review. 2000 Sep 1. Conway JB, Nathan DG, Benz EJ, Shulman LN, Sallan SE, Ponte PR, Bartel SB, Connor M, Puhy D, Wuingart S. Key Learning from the Dana-Farber Cancer Institute’s 10-Year Patient Safety Journey. American Society of Clinical Oncology 2006: 615-619.
Cooperberg MR, Broering JM, Litwin MS, Lubeck DP, Mehta SS, Henning JM, Carroll PR; CaPSURE Investigators. The contemporary management of prostate cancer in the United States: lessons from the cancer of the prostate strategic urologic research endeavor (CapSURE), a national disease registry. J Urol. 2004 Apr;171(4):1393-401. Earle CC, Burstein HJ, Winer EP, Weeks JC. Quality of non-breast cancer health maintenance among elderly breast cancer survivors. J Clin Oncol. 2003 Apr 15;21(8):1447-51. PubMed PMID: 12697865. Epstein R and Street R. Promoting patient-centered communication in cancer settings. Report prepared for NCI, in progress. Esserman LJ. Driving culture change using information and feedback. Presentation at conference, “Advancing the Agenda for Quality in Breast Cancer” January 25, 2006, Boston, MA. Harlan LC, Greene AL, Clegg LX, Mooney M, Stevens JL, Brown ML. Insurance status and the use of guideline therapy in the treatment of selected cancers. J Clin Oncol. 2005 Dec 20;23(36):9079-88. Epub 2005 Nov 21. PubMed PMID: 16301598. Harris Interactive. Misconceptions and Lack of Awareness Greatly Reduce Recruitment to Cancer Clinical Trials. 2006 1(3): 1-3. Gawande A. The Cost Conundrum. New Yorker. June 1, 2009. Institute of Medicine (IOM). 2001. Crossing the Quality Chasm: A new health system for the 21st century. Washington, D.C. National Academy Press. Keating NL, Landrum MB, Guadagnoli E, Winer EP, Ayanian JZ. Surveillance testing among survivors of early-stage breast cancer. J Clin Oncol. 2007 Mar 20;25(9):1074-81. PubMed PMID: 17369571. Leatherman S, Berwick D, Iles D, Lewin L, Davidoff F, Nolan T, and Bisognano M. The Business Case for Quality: Case Studies and Analysis. Health Affairs. 2003 22(2):17-30. Lorig KR, Holman H. Self-management education: history, definition, outcomes, and mechanisms. Ann Behav Med. 2003 Aug;26(1):1-7. Malin JL, Kahn KL, Adams J, Kwan L, Laouri M, Ganz PA. Validity of cancer registry data for measuring the quality of breast cancer care. J Natl Cancer Inst. 2002 Jun 5;94(11):835-44. Malin JL, Schneider EC, Epstein AM, Adams J, Emanuel EJ, Kahn KL. Results of the National Initiative for Cancer Care Quality: how can we improve the quality of cancer care in the United States?. J Clin Oncol. 2006 Feb 1;24(4):626-34. Epub 2006 Jan 9.
McGuire W. Beyond EBM: new directions for evidence-based public health 2005 Perspectives in biology and medicine. 48 (4): 557-569. DOI: 10.1353/pbm.2005.0081 McPherson K, Britton A, Wennberg JE. Are randomised controlled trials controlled? Patient preferences and unblind trials. J R Soc Med 1997; 90: 652-656 Mills EJ, Seely D, Rachlis B, Griffith L, Wu P, Wilson K, Ellis P, Wright JR. Barriers to participation in clinical trials of cancer: a meta-analysis and systematic review of patientreported factors. Lancet Oncol. 2006 Feb;7(2):141-8. Murray A, Montgomery JE, Chang H, Rogers WH, Inui T, Safran DG. Doctor discontent. A comparison of physician satisfaction in different delivery system settings, 1986 and 1997. J Gen Intern Med. 2001 Jul;16(7):452-9. National Academy of Engineering and Institute of Medicine. 2005. Building a better delivery system: A new partnership. Edited by P Reid, WD Compton, JH Grossman and G Fanjiang. Washington, D.C., National Academy Press. Neuss MN, Desch CE, McNiff KK, Eisenberg PD, Gesme DH, Jacobson JO, Jahanzeb M, Padberg JJ, Rainey JM, Guo JJ, Simone JV. A process for measuring the quality of cancer care: the Quality Oncology Practice Initiative. J Clin Oncol. 2005 Sep 1;23(25):6233-9. Epub 2005 Aug 8. Pfeffer J and Sutton R. Hard Facts, Dangerous Half-Truths and Total Nonsense: Profiting from Evidence-Based Management. Harvard Business School Press, Cambridge 2006. Pham HH, Schrag D, Hargraves JL, Bach PB. Delivery of preventive services to older adults by primary care physicians. JAMA 2005 Jul 27;294(4):473-81. Pisano, Gary P., Richard Bohmer, and Amy C. Edmondson. Organizational Differences in Rates of Learning: Evidence from the Adoption of Minimally Invasive Cardiac Surgery. Management Science 47, no. 6 (June 2001): 752. Robinson F, Sandoval N, Baldwin J, Sanderson PR. Breast cancer education for Native American women: creating culturally relevant communications. Clin J Oncol Nurs. 2005 Dec;9(6):689-92. Schachter HM, Mamaladze V, Lewin G, Paszat L, Verma S, DeGrasse C, Graham I, Brouwers M, Sampson M, Morrison A, Zhang L, O'Blenis P, Garrity C. Measuring the Quality of Breast Cancer Care in Women. Evidence Report/Technology Assessment No. 105. (Prepared by the University of Ottawa Evidence-based Practice Center under Contract No. 290-02-0021.) AHRQ Publication No. 04-E030-2. Rockville, MD: Agency for Healthcare Research and Quality. October 2004
Schrag D, Cramer LD, Bach PB, Cohen AM, Warren JL, Begg CB. Influence of hospital procedure volume on outcomes following surgery for colon cancer. JAMA. 2000 Dec 20;284(23):3028-35. Senge P. The Fifth Discipline: The art and practice of the learning organization. New York. Doubleday, 1990. Shortell S, Rundall T, Hsu J. Improving patient care by linking evidence-based medicine and evidence-based management. JAMA. 2007 Aug 8;298(6):673-6. Simone J, personal communication, December 2005. The Breakthrough Series: IHI’s Collaborative Model for Achieving Breakthrough Improvement. IHI Innovation Series white paper. Boston: Institute for Healthcare Improvement; 2003 Wagner EH, Greene SM, Hart G, Field TS, Fletcher S, Geiger AM, Herrinton LJ, Hornbrook MC, Johnson CC, Mouchawar J, Rolnick SJ, Stevens VJ, Taplin SH, Tolsma D, Vogt TM. Building a Research Consortium of Large Health Systems: The Cancer Research Network. J Natl Cancer Inst Monogr 2005;2005(35):3-11. Weeks J, personal communication, December 2005.
NIH…Turning Discovery into Health NIH Publication No. 13-7992 April 2013