Ensuring effective implementation: A fidelity assessment system for comprehensive medication management
Abstract
Measuring the extent to which an intervention is implemented with fidelity (ie, as intended) is critical to its success. Comprehensive medication management (CMM) is an established pharmacy practice intervention in outpatient settings. However, there is no standardized approach to measuring its implementation fidelity. This article describes a fidelity assessment system that includes measures and tools for use by pharmacists and others involved with the practice of CMM. This system is a comprehensive but modular approach to assessing fidelity designed to facilitate measurement along three fidelity dimensions: context (ie, infrastructure needed to support CMM), content (ie, adherence to CMM), and competence (ie, skillset needed to deliver CMM). Practical recommendations with examples are also provided to facilitate application of the system in real-world settings. These recommendations are designed to assist with prioritization of the fidelity dimensions to consider timing of the assessments, use of the resulting data, interpretation of the data, and translation of results into actionable decisions. Incorporating fidelity measurement into any CMM implementation effort is key to ensuring consistent care delivery and impactful clinical outcomes.
1 INTRODUCTION
The misuse, overuse, and underuse of medications contribute significantly to rising health care costs and poor quality care.1, 2 With reliance on prescription drugs as the first line of treatment for most chronic conditions,3 it is imperative that medication optimization interventions, such as comprehensive medication management (CMM), be delivered to maximize impact on patients. In brief, CMM is defined as “a patient-centered approach to optimizing medication use and improving patient health outcomes that is delivered by a clinical pharmacist working in collaboration with the patient and other health care providers.”4 Unfortunately, previous research on the clinical effectiveness of CMM and other medication optimization interventions in outpatient settings has yielded mixed results.5, 6
This lack of conclusive findings has been partially attributed to implementation variability.6 Implementation is defined as “a specified set of activities designed to put into practice a program or service of known dimensions.”7 Implementation variability refers to the variations in service delivery across pharmacists and settings. For instance, adherence to follow-up guidelines after an initial CMM visit might vary depending on the ease with which eligible patients are flagged for follow-up scheduling (eg, access to an automated vs manual flagging process). These differences are a source of heterogeneity that impact the consistency of the clinical outcomes expected to result from implementation of CMM and other medication optimization interventions.
One strategy to reduce implementation variability is to ensure that pharmacists deliver the service as originally intended (ie, with fidelity). Fidelity is defined as the extent to which an intervention is being implemented with conformity to its core components—the elements of the intervention (ie, functions and activities underlying the intervention) that have previously been associated with positive treatment outcomes.8, 9 This is not to say that the intervention should not be adapted to the unique needs of the implementing setting. In fact, contextualization of an intervention (eg, length of a CMM visit, person[s] responsible for following up with the patient) has been found to enhance its uptake.10 However, its core components should be delivered as intended to maximize its potential effectiveness.
Although fidelity is a relatively new concept in pharmacy practice, it has been recognized as a critical precursor of treatment effectiveness in other disciplines (eg, public health, education, mental health).11-16 Interventions that are implemented with fidelity have a higher likelihood of reaching their desired clinical and humanistic outcomes than those that are poorly implemented.8, 11, 13, 17-24 Assessing fidelity alongside the more traditional effectiveness outcomes (eg, medication adherence, clinical indicators) is therefore critical to optimizing medications to improve patient care.
More specifically, assessing fidelity can serve three important purposes. First, fidelity data can be used to gain clarity on negative or ambiguous treatment outcomes. Examining these data allows researchers to determine whether the mixed results obtained in previous studies are due to failures to implement as intended (also known as a Type III error) or to the service itself being effective only under certain circumstances. Second, assuming that the service is effective, monitoring fidelity will facilitate just-in-time adjustments to the service delivery should its implementation deviate over time (known as “implementation drift”). The ability to intervene to avoid drift will facilitate quality delivery, which, in turn, influences the odds of achieving the desired treatment outcomes. Finally, using fidelity assessments will facilitate successful replication and scaling of medication optimization interventions across pharmacists, clinics, and patients. Defining fidelity for a given intervention (eg, CMM) depends on the clear operationalization of the components of the intervention that are critical to its effectiveness. In other words, the development of fidelity assessments hinges on the availability of a usable intervention (ie, a service that has sufficient operational specificity to be teachable, learnable, doable, and readily assessable in practice).25 CMM, for instance, was recently operationalized into five “essential functions” (eg, implement a care plan), with specific associated activities (eg, ensure patient understanding and agreement with the plan), as detailed in “The Patient Care Process for Delivering CMM” report.4 Operational specificity enhances clarity and understanding of the service to be delivered, thereby promoting optimal replication and consistency of implementation across diverse settings and providers.
Assessing fidelity assumes availability of measurement strategies. Presuming that the service is sufficiently defined to be able to establish fidelity criteria, these assessments are typically conducted using one or both of the following methods: (1) expert ratings of documentation, site observations, interviews, and/or videotaped sessions; and/or (2) surveys or interviews completed by the practitioners delivering the service.26 However, examples of fidelity assessments being used in pharmacy practice are few and far between, with these assessments typically being conducted for research purposes.27, 28 For instance, Matzke et al assessed adherence to therapeutic guidelines for identifying and resolving medication-related problems (MRPs) based on a documentation review by a three-member expert panel.29 This fidelity assessment was conducted as part of a research project to evaluate an “Improving Health of At-Risk Rural Patients” care delivery model.
Because fidelity assessments are highly contextualized to the specific intervention, the need to create relevant measurement strategies arises for each unique service. This article specifically describes a fidelity assessment system for CMM that includes measures and tools for use by pharmacists and others involved with the practice of CMM. This proposed system was developed as part of the “CMM in Primary Care” study.29 A joint collaborative effort between the University of North Carolina, the University of Minnesota, and the American Academy of Family Physicians, this study investigated the implementation and effectiveness of CMM across 40 primary care settings with embedded pharmacists. One critical aspect to exploring implementation is understanding fidelity to CMM, hence the need to create the fidelity assessment system described in this paper.
This article aims to: (1) contribute a model for assessing fidelity to CMM; and (2) provide practical recommendations to facilitate application of the system in real-world settings. To our knowledge, this is the first attempt at conceptualizing and operationalizing fidelity for CMM. Practicing pharmacists and others interested in measuring fidelity should be able to use the system and its tools to gain insights into the implementation of CMM. Incorporating fidelity measurement into any CMM implementation effort should produce more consistent outcomes for the targeted patients, and thereby value to payers.
2 THE FIDELITY ASSESSMENT SYSTEM FOR CMM
Our proposed fidelity assessment system for CMM is a comprehensive but modular system that includes practical tools and measures to be used by pharmacists and researchers alike to evaluate and improve selected CMM fidelity dimensions. Its conceptualization is grounded in the National Implementation Research Network's (NIRN) definition of fidelity.7 NIRN operationalizes fidelity along three domains: context, content, and competence. These domains were contextualized to CMM and adapted to ensure coverage of the five traditional dimensions of fidelity (adherence to the intervention guidelines and core components, dose frequency and duration of exposure to the intervention, quality of delivery, participant responsiveness, and program differentiation—identification of unique features of the intervention that are essential to its success [ie, core components]—) (Table 1).13, 30
| Fidelity domains operationalized to CMM | Traditional dimensions of fidelity | |
|---|---|---|
| Context | Availability of infrastructure and processes necessary to support implementation of CMM |
|
| Content | Extent to which the pharmacist and team adhere to the CMM patient care process core components |
|
| Competence | Extent to which the pharmacist and team demonstrate skillful delivery of CMM |
|
- Abbreviation: CMM, comprehensive medication management.
Context refers to the infrastructure, processes, and resources that are necessary to support successful implementation of CMM. Content is the extent to which the pharmacist and team adhere to the intervention guidelines (eg, adherence to CMM core functions and activities). Competence is defined as the extent to which the pharmacist and team demonstrate the skills to deliver CMM successfully. In this study, competence was operationalized as both quality of clinical decision-making and patient responsiveness. Each of the three domains have associated measures and tools that were created as part of the “CMM in Primary Care” parent study, with the exception of quality of clinical decision-making (which was beyond the scope of the funded project) (Figure 1).
Of note is the fact that these instruments are most useful when their application is planned and interpreted by members of a CMM implementation team. Implementation teams are a critical aspect of successful implementation efforts, especially for services that require interdisciplinary collaboration and integration into complex health care environments.31 They are typically composed of three to six staff members (from the clinic setting in which the service is being delivered) with knowledge relevant to a successful CMM rollout (eg, lead pharmacist, clinic manager, other clinical staff such as a nurse practitioner, quality improvement staff, and information technology [IT] staff). Having multiple members of the implementation team review the fidelity data promotes inclusion of multiple perspectives and enables accurate interpretation of fidelity results. The instruments are described below, with excerpts provided in Appendix A.
2.1 Evaluating context: the CMM practice management assessment tool
Screening for and assessing context can be achieved by using the CMM practice management assessment tool (CMM PMAT).32, 33 This tool outlines the infrastructure needed to successfully implement CMM. Examples of concepts included in the tool are organizational support (eg, leadership support), care delivery processes (eg, methods for scheduling patients), and care team engagement (eg, presence and scope of collaborative practice agreements). All of the practice management concepts emerged from grounded theory analysis of interview and focus group data (based on input from 34 pharmacists and 7 CMM managers).33 This instrument consists of three parts. Part I is a general 10-item assessment of all of the CMM PMAT concepts, which respondents rate on a 10-point scale from “less optimal” to “optimal.” Results from Part I help prioritize concepts for in-depth evaluation in Part II. Part II is a 78-item detailed assessment of each of the elements (eg, availability of patient care space) that constitute each concept (eg, organizational support). Part III is a worksheet to facilitate action planning from the results of this assessment. The CMM PMAT is designed to be used by pharmacists and other members of the CMM team to better understand their practice management infrastructure and identify potential areas for improvement. The tool can be used prior to implementing CMM, or at any point during the implementation lifecycle to gain insights into the maturity of an organization's practice management system. Appendix A provides an excerpt of the tool.
2.2 Assessing content: the CMM patient care process self-assessment
To assess content, the project team developed a CMM patient care process self-assessment (CMM PCPSA).34, 35 This survey measures adherence to the CMM patient care process as defined in “The Patient Care Process for Delivering CMM” report.4 As noted above, CMM is operationalized through five “essential functions,” each of which is associated with specific activities. The activities under each of the essential functions in “The Patient Care Process for Delivering CMM” report were condensed into a self-rating assessment. This self-rating assessment is the CMM PCPSA. Respondents are asked to reflect on the last 10 CMM visits, with response options ranging from 0% to 100% of CMM visits on a five-point anchored Likert-type scale (see Appendix A). In addition to adherence, the survey collects information on dose through open-ended questions about the frequency and duration of CMM visits. Results from this assessment can be used to understand strengths and weaknesses in CMM delivery, identify areas for improvement, demonstrate increases in CMM adherence, or simply monitor adherence to CMM. It should be completed at baseline and throughout the CMM implementation lifecycle. This assessment was created based on a methodology that included internal vetting (n = 7), initial testing (n = 42), think-aloud interviews (n = 10), and a pilot test (n = 136). Based on preliminary evidence from the pilot test, this measure was found to be reliable (Cronbach alpha of .97), with each of the essential functions being sequentially linked as intended.34
2.3 Understanding competence: the patient responsiveness survey
Competence can be evaluated by examining both quality of clinical decision-making as a direct assessment of the pharmacist's CMM-related skills, and patient responsiveness as a proxy for quality of service delivery. While developing an instrument to assess clinical decision-making was beyond the scope of this project, a patient responsiveness survey was created and validated.36, 37 Patient responsiveness was defined as the extent to which a patient engages in or accepts CMM, as evidenced by the extent to which CMM met their needs, the quality of the pharmacist-patient relationship, and the overall level of patient satisfaction with their CMM experience. This 30-item survey was designed to capture the patient experience while evaluating the clinical pharmacist's ability to build a relationship with the patient around medication optimization. Patients are asked to rate each survey item (eg, my clinical pharmacist provides useful recommendations on how to take my medicines) using a four-point anchored Likert scale, from “strongly agree” to “strongly disagree” (see Appendix A). The instrument was developed and validated using a multiphase process including internal vetting (n = 7), and initial pilot testing with 128 patients served by 32 pharmacists. Based on the pilot test, the survey demonstrated excellent reliability (Cronbach alpha of .98), with good content validity and preliminary evidence of construct validity supporting a one-factor structure.36, 37 The survey can be used as part of a fidelity assessment to evaluate quality of service delivery based on the patient CMM experience.
3 RECOMMENDATIONS FOR USING THE FIDELITY ASSESSMENT SYSTEM FOR CMM
The following recommendations reflect the best practices in the fidelity literature,7, 26, 38-45 as well as the authors' insights and experience with the system as part of the parent study. These insights were collected during the development and validation of each tool. The recommendations below are intended to facilitate application of the fidelity assessment system.
Recommendation 1: Although the assessment system is designed to measure fidelity as a multifaceted concept, it is acceptable to select the fidelity domain and tool(s) most relevant to your implementation efforts.26, 38
CMM is an intervention that requires adherence to a defined process, clinical knowledge and skills, relational skills for working with patients and other health care providers, support systems and infrastructure, and coordination with multiple team members and patients. With this level of intervention complexity, using multifaceted assessments to evaluate fidelity to the diverse aspects of the service is highly recommended. Ideally, a clinic should explore each of the three fidelity domains to understand the delivery infrastructure (context), the intervention components (content), and the skill set required to deliver the intervention (competence).
However, it is not always realistic or necessary to evaluate all three domains or use the full set of tools that are part of the CMM fidelity assessment system. Selection of the most relevant fidelity domain or appropriate assessment tool(s) should be guided by the organization's needs, priorities, and resource limitations. For example, a health care system is hiring multiple CMM pharmacists who just completed residency training at the institution. Their pharmacy manager has access to preceptor observations during residency, in addition to making personal observations of their patient visits. He realizes that the CMM pharmacists' residency preceptors held them to the same high standard in carrying out CMM as current team members. In this case, the manager may feel more confident about their adherence to the CMM patient care process. Measuring content fidelity might therefore be a lower priority.
The decision to select a particular fidelity domain or tool rather than the full system has its limitations. For instance, a more focused assessment will limit data interpretation. If evaluating only one domain, the resulting scores will not reflect fidelity to the CMM service overall, but rather fidelity in that particular domain. Data should be cautiously interpreted as to not overextend and draw conclusions beyond the domains measured or tools used. If the focus of the assessment is content fidelity, higher adherence to the patient care process indicates that CMM is being delivered as operationalized in “The Patient Care Process for Delivering CMM” report.4 It does not suggest that the patients are adequately engaged with CMM and satisfied in their interactions with their pharmacist (ie, competence fidelity); nor does it demonstrate that the infrastructure needed to successfully implement CMM is adequate (ie., context fidelity). Using the full fidelity assessment toolbox is recommended to maximize its benefits. However, it is also acceptable to be selective based on resources and priorities provided careful attention is paid to data interpretation.
Recommendation 2: Timing of the fidelity assessment should be determined based on the stage of CMM implementation and the purpose of the assessment.39, 40
Best practice encourages use of fidelity assessments in an ongoing, cyclical way rather than as a one-time measurement strategy.26, 40, 46 The timing of the assessment will most likely be contingent on the stage of CMM implementation and the goals of the assessment. Implementation stages refer to phases of implementation over time, such as pre-implementation, initial implementation, full implementation, and stabilization.47 Although they are often non-linear, these stages are associated with a particular set of implementation strategies and activities, including specific uses and goals for fidelity assessments. For instance, when implementing a new service, the purpose of the fidelity assessment should be to establish the needed CMM infrastructure and ensure consistency of CMM delivery to maximize potential benefits for patients. In this case, it might be beneficial to conduct more frequent assessments until the desired level of fidelity is achieved (eg, quarterly). After initial implementation of CMM, fidelity measurement may be used to monitor potential “drift” in CMM delivery or identify areas of improvement. If the goal is to monitor “drift,” it might be sufficient to assess fidelity every six to 12 months. If fidelity data are used to track positive changes for improvement purposes over the short-term, more frequent assessments might be warranted. Finally, once CMM has been implemented for multiple years, it can be useful to continue monitoring overall fidelity periodically to ensure successful sustainability and scaling.
It is important to recognize that fidelity is zero prior to the launch of a new intervention. Therefore, fidelity measurement should occur only after practitioners start delivering the service. For organizations that attempt to measure CMM fidelity in the early stages of implementation, it is common to see higher than expected initial fidelity scores.48-50 These results can be attributed to a number of factors, including overestimation of practitioners' ability to carry out the intervention as intended and drift having not yet occurred because training would have been recently completed. Consequently, it is not atypical for fidelity scores to decrease (rather than increase) from baseline to later repeated measurements as the service continues to be implemented. Ultimately, a longitudinal approach that takes into account the implementation stage and purpose is best for accurately measuring fidelity.
Recommendation 3: Fidelity data should be used constructively, not punitively.7, 41
It is strongly recommended that information from the fidelity assessment be used to improve practice, or as part of research. It was not designed to produce data to make evaluative decisions based on performance. In keeping with the philosophical principles of implementation science, fidelity assessments are intended to be used constructively, not punitively. CMM providers need to feel sufficiently comfortable to adopt a growth mindset in approaching the fidelity assessment without wondering if the quality of their clinical skills or the care they provide is in question.51 Self-reported data are always at risk of influence by social desirability bias. If practitioners are additionally concerned about use of the data, their responses to the assessment will almost surely be impacted. Without honest and accurate data, the usefulness of the fidelity assessment system becomes severely limited, both from a practice and research perspective.
For instance, a CMM pharmacist may naturally be hesitant to report that he does not consistently follow-up with patients, even when constructive use of the data has been made clear. The accuracy of his responses might be colored by overestimating other providers' success rates with follow-up, wanting to provide answers he thinks others want to hear, or not wanting his peers and colleagues to perceive him as providing inadequate care. Having the pharmacy manager and clinic leadership promote a culture of improvement and growth, and emphasize the need for honest self-reflection, will encourage the CMM pharmacist to use the fidelity assessment as an opportunity to optimize patient care.
Obtaining consensus on constructive use of the data also assumes data dissemination issues have been considered. If results are to be shared with leadership or externally, it is important to determine how the data will be aggregated or de-identified to avoid connection with individual sites or practitioners. Should individually identifiable responses knowingly be disseminated, it will most likely impact the willingness of CMM providers to participate in fidelity assessments or honestly reflect on their CMM practice. In summary, the usefulness of the fidelity assessment will be influenced by effective communication related to its purpose as a constructive tool.
Recommendation 4: To be meaningful, data will need to be interpreted within the unique health care environment from which it is collected.40, 42, 43
Meaningful interpretation of fidelity data always requires some level of contextualization. The results need to be analyzed through the local lens of the health system, clinic, and practitioner. The populations served, goals of the organization, and numerous other factors may influence the conclusions that can be drawn. For example, after having her patients complete the CMM patient responsiveness survey as part of the fidelity assessment, one pharmacist notes that her patients' scores were significantly lower compared to those of two of her colleagues. The logical conclusion would be that she is simply not as engaged with her patients as the other pharmacists. However, upon closer examination and discussion of these results, it turns out that she is responsible for caring for an uninsured transient patient population with low levels of health literacy. Her colleagues, on the other hand, serve a more affluent, commercially insured population. In this case, the differences in patient responsiveness scores might be influenced by the patient population served by the pharmacist.
Without contextualizing interpretation of the data, there is a risk of making erroneous conclusions. To mitigate this risk, it is recommended that the implementation team members review fidelity results together. Discussing results as a team will help ensure that all potential explanations are explored and that multiple perspectives are considered when interpreting fidelity information.
Recommendation 5: To be useful, fidelity data need to be translated into actionable strategies.44, 45
Regardless of the reason for conducting a fidelity assessment, the data will be useless unless they are translated into meaningful and actionable strategies and insights. The translation process should at a minimum involve review and interpretation of the information (see Recommendation 4), prioritization of the most relevant findings and concerns, generation of potential solutions and strategies to address these concerns, and prioritization of the solutions that are most likely to lead to the desired changes. This process should culminate in an action plan that the implementation team is responsible for executing. This action plan should include the desired change, SMART goals (specific, measurable, achievable, realistic, and time-bound), specific actions to attain these goals, a timeline, accountable parties, and strategies to measure success.
For example, a clinical pharmacist who has been delivering CMM for a couple of years has just reviewed “The Patient Care Process for Delivering CMM” report.4 He decides to make sure that CMM is being implemented as intended at his clinic, and has all of the pharmacists complete the CMM PCPSA for the first time. The results highlight multiple opportunities for improvement. After reviewing and discussing the assessment scores, the implementation team decides to prioritize its improvement efforts on increasing patient follow-up. Patient follow-up seems to be implemented with the most variability across all of the pharmacists at this clinic. The team identifies a number of reasons for this gap, one of which is the lack of a process to identify patients who need follow-up care. Their action plan includes the development and implementation of a new protocol that has the pharmacist flag the patient who needs follow-up and sends an electronic message to the medical assistant letting them know the timeframe and reason for follow-up. With their SMART goal being that 100% of patients needing follow-up appointments have those scheduled within the appropriate timeframe, they also include testing of the new protocol over a three-month period to assess the effectiveness of their solution. If the change is successful, the team would expect the essential function 5 (follow-up) of the CMM PCPSA to be higher post-implementation of the protocol.
There are several factors to consider when translating findings and executing an action plan, including potential resistance to change, levels of leadership support, knowledge and skills that are needed to carry out the change, and access to resources. Action planning does not only require translation of data into an action plan, it also involves assessing contextual facilitators and barriers. Taking the time to discuss how to capitalize on facilitators, while brainstorming and addressing potential challenges prior to execution will increase the likelihood of successfully using fidelity data to make positive changes.
4 DISCUSSION
Application of fidelity assessments is critical to ensuring that services are implemented as intended. Implementation fidelity, in turn, facilitates achievement of treatment effectiveness outcomes. Because fidelity assessments are highly contextualized to each service, it is necessary to create measures and tools that are specific to CMM. Although CMM is an established intervention in outpatient settings, there is no standardized approach to measuring its implementation fidelity. This article describes a fidelity assessment system with practical assessments and tools, and provides recommendations to facilitate its application. To our knowledge, this is the first published article describing a comprehensive yet practical CMM fidelity assessment system for use in practice and research. In addition, although this system is specific to CMM, it can serve as a conceptual foundation for measuring fidelity to other medication optimization interventions.
As an early attempt to standardize the measurement of fidelity to CMM, the assessment system is not without its limitations. First, although preliminary pilot testing has supported the reliability and internal validity of the proposed tools and surveys, additional research efforts are needed to further establish their psychometric properties, especially construct and external validity. Second, the instruments included in the assessment system were developed for use in primary care clinics with embedded pharmacists. Use in other types of settings (eg, community pharmacies, specialty clinics) would most likely require some modifications, especially to the CMM PMAT. For instance, in a teaching clinic, methods for identifying patients in need of CMM may not be considered essential, as pharmacists focus on patients being cared for by learners rather than the highest risk patients. Third, assessing fidelity is an investment. It requires time, staffing, and other resources that busy practices might not be ready to commit. In this case, in addition to a limited application of the fidelity assessment system (Recommendation 1 above), it is important to obtain buy-in from leadership by highlighting the benefits of measuring fidelity. Finally, the system surveys and tools primarily rely on self-report data. This assessment method has a number of advantages (eg, ease of administration) and is commonly used to evaluate fidelity.26 However, it is also imperative that measures be taken to minimize its drawbacks, such as social desirability. In addition to encouraging constructive use of data (Recommendation 3 above), triangulating results with objective data will strengthen validity and confidence in the findings. It is worth noting that completion of the CMM PCPSA survey was initially designed to be supplemented by objective ratings of CMM using pharmacists' clinical documentation (eg, SOAP notes). Unfortunately, clinical documentation was found to be highly variable across care delivery settings and therefore insufficiently reliable to provide an accurate description of CMM content adherence.52, 53 This is not to say that clinical documentation cannot be used as an additional source of fidelity data. However, this would require the development and use of a formal structured documentation format, with clear guidelines around the content to be included. Viable sources of data (eg, site observations) that can augment the current fidelity assessment system need to identified and validated in the future.
In moving towards value-based health care, it is imperative that pharmacists demonstrate the clinical, humanistic, and economic impact of CMM. This goal will be only be achieved by expanding the study of CMM and other pharmacist-provided services beyond traditional effectiveness outcomes to also include implementation fidelity. Advancing pharmacy practice requires a paradigm shift, one in which it is no longer acceptable to ignore the black box of implementation.
ACKNOWLEDGMENTS
The authors gratefully acknowledge all of the pharmacists and primary care practices engaged in the parent study for their valuable work and insights. In addition, this work would not have been possible without the contributions and insights provided by the University of North Carolina and the University of Minnesota “CMM in Primary Care” study research team members. Finally, the authors acknowledge the generous support for this study provided by the American College of Clinical Pharmacy (ACCP) and the ACCP Research Institute.
CONFLICT OF INTEREST
The authors declare no conflicts of interest
FUNDING INFORMATION
Funded by the American College of Clinical Pharmacy (ACCP) and the ACCP Research Institute as part of the Enhancing Performance in Primary Care Medical Practice through Implementation of CMM grant.
APPENDIX A: CMM Fidelity Assessment System Tools and Measures
Appendix A contains brief excerpts from the following instruments: the comprehensive medication management (CMM) practice management assessment tool, the CMM patient care process self-assessment, and the patient responsiveness survey.
CMM practice management assessment tool (CMM PMAT)
This tool is used to assess the infrastructure needed to support CMM, and identify areas for improvement. This tool consists of three parts: Part I, a general 10-item assessment of all of the practice management domains that is used to prioritize areas for further assessment in Part II; Part II, a 78-item detailed assessment tool of all domains and associated components; and Part III, a worksheet to facilitate action planning for practice management improvement. Brief excerpts are provided below. 
CMM patient care process self-assessment (CMM PCPSA)
This survey is designed to assess the degree to which CMM is implemented as intended. The pharmacist is asked to reflect on the last 10 CMM visits and select the most appropriate category for each activity. Response categories represent the percent range of CMM visits during which they completed a specific activity. Example items for two of the five essential functions assessed are provided below. 
CMM patient responsiveness survey
This survey is intended to be administered to patients after they have had at least one CMM visit with a clinical pharmacist in a primary care physician office. Patients are asked to complete the following questions. Response categories range from “Strongly Disagree” to “Strongly Agree” on a 4-point Likert scale. Below are example items from the survey. 
