Below you can browse the online version of the MULTI-ACT Manual or download the Manual and the appendices for offline use.
Please note that you can browse the Master Score (Appendix 4) also using the Toolbox functionality.
· MULTI-ACT Complete Manual (pdf)
· MULTI-ACT Manual Appendix 1 (pdf)
· MULTI-ACT Manual Appendix 2 (pdf)
· MULTI-ACT Manual Appendix 3 (pdf)
· MULTI-ACT Manual Appendix 4 (xlsx)
· MULTI-ACT Manual Appendix 5 (pdf)
1.1 Structure of the CRIF Manual
1.1 For whom the CRIF Manual is intended
1.2 How to use the CRIF Manual
3 COLLECTIVE RESEARCH IMPACT FRAMEWORK
Engagement Coordination Team (ECT).
Stakeholder Advisory Board (SAB)
Committees and Working Groups (WGs)
Criterion 1: Mission and agenda
Criterion 2: Participatory Governance
Criterion 3: Clear, effective and inclusive methodology of stakeholder engagement
Criterion 4: Effective and efficient management and coordination of the initiative
Criterion 5: Co-accountability assessment
5 PATIENT ENGAGEMENT STRATEGY AND GUIDELINES
5.3 Science of/with patient input
5.4 Return on Engagement (RoE)
5.5 Patient Engagement Roadmap
Roadmap Action 1: Establishment of an Engagement Coordination Team (ECT)
6 COLLECTIVE IMPACT ASSESSMENT
Core and additional indicators
Qualitative and quantitative indicators.
6.3 Patient Reported Outcomes (PROs) and Patient Reported Outcomes Measures (PROMs)
Identification of stakeholders
After clicking the “Perform Materiality Analysis” button (1) and saving, you go the Admin Panel (2).
7.7 Impact Assessment Dashboard: the Master Scorecard
PARTNERS OF THE MULTI-ACT CONSORTIUM
THE CRIF MANUAL
This Manual collects all the key outputs of the MULTI-ACT project. which are integrated in the Collective Research Impact Framework (CRIF). The final version of the CRIF, after validation and refinements made following a case study, is presented hereafter. The case study’s focus was a research and innovation multi-stakeholder initiative focused on multiple sclerosis treatment and care. The CRIF Manual consolidates all components of the framework to make it easy to use for the research and innovation organisations like yours.
1.1 Structure of the CRIF Manual
The CRIF Manual’s structure is intended to first introduce you to the Collective Research Impact Framework (CRIF) and its main concepts (chapter 2: About the MULTI-ACT project), and then present more detailed information and guidelines in subsequent chapters dedicated to individual CRIF parts. The chapter 3: Collective Research Impact Framework explains the CRIF’ underlying philosophy, structure, building blocks, tools and the recommended pathway of implementation. The chapter 4 presents full Governance Criteria, and chapter 5 – Patient Engagement Guidelines. In the chapter 6 Collective Impact Assessment, you will explore the Materiality Analysis and Master Scorecard. Lastly, the chapter 7 Toolbox is a brief introduction to the web-based platform, which we encourage you to use in the CRIF implementation process from the very start.
1.1 For whom the CRIF Manual is intended
This Manual is addressed mainly to the organisations and individuals who are interested in or responsible for implementation of the CRIF in responsible research and innovation (RRI) initiatives in the area of brain research and health research in general. The CRIF Manual’s purpose is to guide these organisations through the process of adoption of the CRIF.
1.2 How to use the CRIF Manual
The CRIF’s main goal is to help your initiative in applying RRI principles and making a positive social impact by creating a collaborative, participatory process among your different stakeholders.
We recommend that you go through the CRIF Manual first and familiarize yourself with the components of the CRIF, its concepts, and its terminology. Doing that will make it easier not only to implement the steps advised in the CRIF Manual itself but also to use the Toolbox tools more efficiently.
The Toolbox is an indispensable companion of the CRIF Manual and you will find many cross-references between them. In the CRIF Manual, there will be basic instructions on how to use the Toolbox but it is intuitive and easy to follow on its own. You can set up an account even now (https://toolbox.multiact.eu), before reading the whole Manual. It is free to use.
The Manual illustrates the rationale behind collaborative governance and stakeholder engagement methods and tools, such as the Baseline Analysis (BA) and Materiality Analysis (MA), that can only be performed via the Toolbox. The results of the Baseline Analysis will serve to profile your research multi‑stakeholder initiative and suggest which parts of the Governance Chapter you should pay special attention to. With the help of the Patient Engagement Plan tool, your initiative will be able to choose the best methods for patient engagement and smoothly organise the process appropriately for every stage of the research. Materiality Analysis will determine which impact aspects are most relevant for the stakeholder involved in your research and consequently propose you a set of indicators that your initiative can use to build its tailored impact scorecard and evaluate its impact in a co-accountable manner. In the case of Materiality Analysis, you can use the Toolbox to engage your initiative’s stakeholders in such a process with a “sending invitation” function.
We developed this document on the assumption that CRIF should be flexible and customizable, so you will find out that many activities are left to your discretion: you should use them according to your specific situation, needs and your best judgement also in consideration of the stage your initiative is currently in.
2 ABOUT THE MULTI-ACT PROJECT
MULTI-ACT. Collective Research Impact Framework and multi-variate models to foster the true engagement of actors and stakeholders in Health Research and Innovation (https://www.multiact.eu/) is an EU-funded project with a goal of increasing positive impact of health research on the society by applying a Responsible Research and Innovation (RRI) approach. It has created the Collective Research Impact Framework (CRIF). The CRIF offers a participatory and realistic evaluation of impact of health Research and Innovation (R&I) multi-stakeholder initiatives through:
· 
Governance Criteria which facilitate cooperation of all relevant stakeholders in defining the mission and agenda for health research initiatives, while ensuring participative, patient‑focused and efficient operation.
· New metrics for the evaluation of the research results to enable multi-dimensional impact assessment and thus overcome the limitation of the current focus on research excellence
· Comprehensive patient engagement guidelines to foster their effective involvement in research programmes and projects in line with the core objectives of the “Science with and for Society” (SwafS) H2020 programme and specifically its ambition to enable public engagement in RRI.
The MULTI-ACT project works with patients and patient organizations, research organizations, academics, policy makers, neurologists and other care providers, scientists and pharmaceutical industry to develop innovative tools that will help you assess the collective impact of your research, implement the best governance practices and incorporate experiential knowledge of the engaged patients and their communities.
2.1 Project rationale
179 million Europeans will be affected by a brain disorder at some point in their lives: an estimated 1 in 3. In 2017, 307,9 million brain diseases were counted in the 28 European Union member states (EU28), of which 74,5 million were newly diagnosed, including Alzheimer’s disease and other dementias, epilepsy, headache (migraine and tension-type headache), multiple sclerosis, Parkinson’s disease, brain cancer, motor neuron diseases, neuroinfectious diseases, and stroke (Gustavsson et al., 2011; Deuschl et al., 2020). The WHO stated that brain disorders account for 35% of the burden of all diseases in Europe (Wittchen et al., 2011).
Patients with brain disorders had a total number of disability-adjusted life-years (DALYs) of approximately 21 million and the total number of deaths was 1,1 million. After cardiovascular diseases and cancer, neurological disorders’ burden on DALYS and deaths in the member states was the third highest. The proportion of total DALYs attributable to neurological disorders was approximately 13,1% and the proportion of deaths was around 19%. In addition to the negative impact on healthy life years and the quality of life, brain disorders also have consequences beyond the healthcare system by impacting the increasing costs of technological progress, prolonged impairment, great dependency and significant reduced productivity at work, as well as the burdens on health and social welfare systems (European Brain Council, 2017; Deuschl et al., 2020).
The annual direct and indirect costs for the EU economy and national health budgets of these disorders exceed 800 billion euro, of which 60% is attributable to direct healthcare and non-medical costs and 40% is from the loss of productivity in the labour market. The average yearly costs of brain disorders per person vary considerably from one disease to another based on the severity of and life expectancy with the disease. Whereas a person with chronic headaches incurs on average 285 euros per year, someone with Multiple Sclerosis incurs approximately 27.000 euros per year on average (Gustavsson et al., 2011; European Brain Council, 2017).
The rise in the number of people with brain diseases (such as Alzheimer’s, Parkinson’s, depression, Multiple Sclerosis, addictions, and many more) and the high proportion of deaths and DALYs attributable to those diseases are due to factors such as higher life expectancy and the increasing incidence and the increasingly long duration of diseases related to ageing. There are substantial sex differences in the burden of neurological disorders within the member states. DALY rates for dementia, migraine, and multiple sclerosis were higher in women in all age groups, whereas the rates were higher for men as they relate to stroke and Parkinson’s disease. The sex differences reflect the differing distribution of each clinical condition in men and women (Deuschl et al., 2020).
It is of utmost importance to develop a mission-oriented research model that, in addition to producing academically excellent research, addresses societal concerns and delivers results that have a real impact on the lives of affected patients and their caregivers.
To mitigate the burden of brain disorders, research and innovation initiatives must become more collaborative and co-accountable. So far, most multi-stakeholder initiatives have lacked an impact assessment system shared among its stakeholders. A support infrastructure which would ensure alignment of efforts and accountability has been also missing (Zaratin, Battaglia and Abbracchio, 2014; Zaratin et al., 2016).
In the past decade, many collaborative research initiatives were launched with the view of developing innovative treatments for brain disorders. Despite the significant progress in terms of understanding the mechanistic underpinnings of neurological diseases at the molecular, cellular and circuit levels, translation of these discoveries into therapies remains a critical challenge.
Taking patients’ needs and perspectives into account through the entire research process is another challenge research initiatives encounter. Aligning differing priorities and assessment systems of the members of the research initiatives is another. Cooperation among various organizations is often identified as a key success factor in maximizing the positive impact of research and innovation initiatives in the brain disorders area. Different stakeholders need a shared language and shared metrics to be able to be accountable to one another and progress towards the mission.
Fostering Responsible Research and Innovation requires different stakeholder groups’ commitment to find collective solutions to solve a specific problem (mission) and, thus, achieve a socially desirable result (von Schomberg, 2013) through the fulfilment of a number of strategic objectives (agenda). In health research and innovation, it entails collaboration of academia, government and regulatory agencies, patients’ and citizens’ organisation, healthcare organisations, biotechnological companies (biotech), pharmaceutical companies and others along the entire research and innovation research pathway.
In MULTI-ACT, we departed from research on multiple sclerosis as the basis to develop the proposed Collective Research Impact Framework (CRIF).
One of the key novelties it entails is its multidimensional approach to assess research impact that integrates conventional metrics related to research excellence with new ones, relating to economic impact, efficacy (intended as adherence to the common mission), social impact and patient-reported outcomes. We also conducted extensive consultations (e.g. surveys, interviews) with several stakeholder representatives, such as healthcare professionals, patients, policy makers and industry actors. On this basis, we formulated recommendations on how and when to engage patients to allow them to contribute their most valuable experience and opinions. During the project, the CRIF was tested and adapted for initiatives doing research on other brain diseases as well.
3 COLLECTIVE RESEARCH IMPACT FRAMEWORK
In the context of multi-stakeholder initiatives, accountability is a relationship among stakeholders who are required to give account for their actions. Traditionally, accountability was addressed to shareholders and concentrated on financial results and processes. Nowadays, multiple categories of stakeholders both need to be consulted and reported to: not only shareholders, but also other stakeholders: customers, employees, local community, NGOs etc. In response to this shift in accountability relations, and even higher complexity of multi-stakeholder initiatives, MULTI-ACT puts forward the concept of co‑accountability: it is a democratic and participatory approach to implementing accountability that incorporates plurality of stakeholders’ perspectives into decision-making processes, while recognizing their competing and complementary interests around health research. Co‑accountability is the theoretical foundation of the CRIF. All CRIF tools have an objective of establishing co‑accountability among stakeholders. Its model enriches and evolves the Integrated Accountability Model (IAM) (Andreaus and Costa, 2014). In order to accommodate all the impact dimensions which we deemed most relevant for a comprehensive impact assessment of health R&I, we added two more dimensions, i.e. excellence (scientific and academic quality) and patient‑reported dimension, to the three dimensions proposed in the IAM (efficiency, mission fulfilment (efficacy), and social impact).
The five CRIF dimensions, and the corresponding impact aspects and indicators proposed to assess them, cover all the most relevant areas of impact of brain research. Thanks to this, you can be sure that your initiative assesses its impact in a comprehensive, holistic way. At the same time, CRIF’s impact assessment is flexible – you do not need to use all 125 indicators as long as you use a minimum number from each dimension. Your initiative’s stakeholders help you choose which aspects to measure. This allows your initiative to establish priorities, monitor progress, report the results and – last but not least – talk about your achievements in a language relevant to all key stakeholders.
In addition to facilitating internal and external communication, CRIF helps your initiative’s many stakeholders unite around common goals despite their competing interest. The Governance Model provides guidelines on how to engage stakeholders to formulate a common mission and agenda. Stakeholder engagement, and especially patient engagement, should permeate all management operations. Moving to a more open, co-creative approach, as well as changing the focus of the analysis from an organization’s objectives to the social issue unifying the field, CRIF enables deeper analysis of the relationships established between different stakeholders. In summary, the CRIF gives you tools to:
· Engage stakeholders – initiative’s participant organizations, patients, their families, and caregivers;
· Subsequently, involve these stakeholders in selecting the metrics that all the initiative’s participants will employ for assessing their collective impact and monitoring their performance;
· Use multidimensionality in its co-accountability approach by measuring impact in five areas, i.e., efficacy in reaching the mission, efficiency in economic and financial performance, research scientific excellence, broad social impact, and – last but crucial – patient-reported perspective.
· Offer a principle-based, participatory governance model which makes it possible to implement the RRI approach.
The CRIF is intended for organizations grouped in multi-stakeholder initiatives working on or willing to start conducting their R&I in the area of brain disorders. Though, it is conceived to be flexible and extensible to other health research domains. These organizations should be interested in adopting a multi-stakeholder, participatory approach based on co-accountability and focusing on reaching their transformational mission.
CRIF is also designed to meet requirements set for Responsible Research and Innovation (RRI), which must (Strand et al., 2015; Yaghmaei, 2018):
· Include stakeholders;
· Make researchers and societal actors mutually responsive;
· Strengthen the relevance of ethical standpoints and sustainability in decision-making;
· Improve the outcomes and maximising the impact of research.
There are many reasons to adopt the CRIF. For many initiatives, the Framework will offer methods of implementing what they already had in mind and what they believed in: genuine patient engagement, participatory governance, and ability to evidence impact. Additionally, CRIF helps to ensure continuity of research initiative by promoting stakeholders’ commitment, and financial sustainability. All these qualities may help to meet stringent requirements of the funding agencies, whether private or public, and make the initiatives compliant with CRIF good candidates for other projects.
Being a flexible and customizable framework, the CRIF does not place an undue burden on its Appliers, allowing them to focus on what matters most to the final Beneficiary of research: patients and the society.
3.1 Co-accountability Pillars
Health research impact is a complex phenomenon. To measure it, perspectives and values of different stakeholders engaged in the research need to be understood and integrated.
The MULTI-ACT CRIF represents a valuable step forward in this direction as it makes stakeholder engagement the backbone of the process.
Co-accountability Pillars are one of the ways to conceptualize the implementation process of the CRIF, during which the stakeholders’ perspectives and values are gathered and integrated into the evaluation tools of research initiative. We created them based on the analysis of the most relevant state-of-the-art impact assessment methodologies and refined them during consultations with stakeholders. They describe the flow of the collective impact assessment process, expressing the philosophy of the CRIF.
|
|
Mapping of stakeholders and establishment of the scope |
Based on the mission, the research initiative will select the stakeholders, which are engaged in setting or refining the agenda that the research initiative aims to achieve. The research initiative should identify the potential stakeholders that are strategic in the fulfilment of the impact. In defining the priorities, the plurality of interests should be considered, according to the CRIF dimensions (efficacy, excellence, social, economic, patient-reported). |
|
|
Development of operative framework |
Stakeholders are engaged in defining the resources, activities and desired results. The governance model should be agreed together with the stakeholders and aligned with the different perspectives related to the dimensions of CRIF. |
|
|
Co-selection of aspects |
Stakeholders are engaged in identifying the most relevant aspects for mission of the initiative. In the selection, multiple aspects related to all the dimensions of CRIF should be ensured. |
|
|
Shared measurement system |
Stakeholders are engaged in data collection, analysis, co-selection and customization of indicators. The measurement system should enable a multi-perspective approach: with the Master Scorecard, the impacts are assessed from the multiple perspectives considering the dimensions of CRIF. |
|
|
Reporting, monitoring and assessment
|
To facilitate collective decision making, the results should be reported and monitored for each dimension of CRIF. The impact assessment supports the shared mission enabling refinement of the activities to increase the impact on people and society. This pillar represents a starting point for the whole process, thus making co-accountability a dynamic and iterative process. Therefore, this pillar represents both the end point and the starting point of the process, because the iterative process allows learning and continuous improvement. |
Table 1 Co-accountability Pillars description
The Co-accountability Pillars represent two key features of the CRIF:
· Circularity: an on-going engagement process and re-definition task within the research initiative. Circularity guarantees a dynamic and an iterative approach.
· Strategic value: they offer a possibility to adapt and assess research initiative through continuous monitoring.
3.2 CRIF components
The MULTI-ACT CRIF relies on three main conceptual components i.e., the Governance Criteria, the Patient Engagement Guidelines and the Master Scorecard. The first two took a form of guidelines and recommendations describing what to do and how to do it. The Master Scorecard is a set of 125 indicators intended for monitoring and reporting.
These components are accompanied by a digital Toolbox with functionalities for stakeholder engagement, analyses, and impact assessment. Namely, it allows to perform the Baseline Analysis and the Materiality Analysis and it accompanies the initiative owners in the design of their Patient Engagement Plan and their tailored Master Scorecard. This Manual provides guidelines for all other tools.
You will find that all the elements of the CRIF are intertwined: after having conducted the self‑assessment exercise meant to profile your initiative (Baseline Analysis), the governance process requires conducting Materiality Analysis and implementing a common agenda and measurement system, which are in turn instrumental to enable Co-accountability. As a result, all the indicators point towards achieving the mission and agenda formulated at the early stages of the governance implementation.
Governance Model
The Governance Criteria are a set of recommendations on how to organize your initiative’s governance bodies, define its mission and agenda, and implement a monitoring and measurement system. Thanks to the Governance Criteria, your initiative can define its mission and shared agenda in accordance with the MULTI-ACT principles of stakeholder engagement and co-accountability. You will also find instructions how establish a shared and effective assessment system, including a set of indicators of the Master Scorecard that promotes improvement and communication, and set a mechanism to receive feedback.
They facilitate collaboration among different stakeholders and improve stakeholder engagement. The model is developed according to the Responsible Research & Innovation (RRI) agenda, which aims to encourage societal actors to work together to better align research and its outcomes with the values, needs and expectations of society.
The Governance Model includes 5 Criteria and 19 sub-criteria detailed in 41 recommendations. The Criteria are not rigid steps to be followed, rather they are meant as general requirements to be met. Baseline Analysis is a web-based questionnaire, a part of the Toolbox, which you can use to assess your initiative’s compliance with the Governance Criteria.
The implementation of the Governance Criteria guarantees an inclusive and equitable governance model, which allows the involvement of all interested parties under a co-design approach. It helps you put in place comprehensive, balanced and efficient stakeholder engagement process, ensuring also the participation of patients, their families and care givers, and patients’ organizations. Finally, it promotes an effective, cooperative and efficient coordination and alignment of the objectives and actions required to pursue the vision and the agenda of the initiative.
While developing the Model, we considered both the practical solutions implemented by existing multi-stakeholder initiatives from various health and non-health sectors and the recommendations emerging from a context analysis, and the approach and objectives of the MULTI-ACT project itself, namely fostering the diversification of stakeholders in Health Responsible Research and Innovation processes. We looked into various collaborative multi-stakeholder initiatives and their governance systems and best practices, paying special attention to MULTI-ACT’s principles i.e., developing a participatory governance model, co-designing a transformational agenda and adopting a co-accountability approach.
Patient Engagement Guidelines
The Patient Engagement Guidelines are an operative guide for meeting the criteria “participatory governance” and “effective stakeholder engagement” for the key and often under-represented stakeholder category “patient, their families and caregivers”. The Patient Engagement Guidelines provide advice on how to engage patients and to what extent to include them in your decision-making processes depending on your situation. They will help you select the research priority and stages of research where patient engagement is instrumental to meet the initiative's mission and agenda.
The actions covered by the guidelines include:
· Establishment of governance bodies in charge of patient engagement (Criterion 2: Participatory Governance) and training of its members, from recruitment to cooperation with other bodies,
· Formulation of appropriate plans for patient engagement (Patient Engagement Plan) for each identified research priority & step,
· Choosing from a catalogue of methods for stakeholder engagement (Criterion 3: Clear, effective and inclusive methodology of stakeholder engagement) and finally
· Monitoring and assessing the impact of patient engagement.
Patient engagement strategies are directed to engage patients according to specific needs and requirements that emerge on each of the 7 stages of research, described by Research &Innovation Path. Engaging patients both in the governance of research & innovation (Science with Patient Input) and in the impact assessment (Science of Patient Input) is instrumental to meeting transformational mission’s health R&I. High-standard patient engagement strengthens credibility and improves research results. Considering that the whole society is going to be “patient, family or caregiver” in some periods of the lifetime, it also makes it easier to maximize research social impact.
The guidelines also offer a set of patient-reported outcomes indicators to measure the success and effectiveness of this engagement. The value and effectiveness of patient engagement relies on producing outcomes that matter to patients, while being financially sustainable in achieving this goal.
Over the last decade, along with the democratization of health sciences and patients’ empowerment, patient engagement has become increasingly important. Patients have been actively engaged as co‑researchers and can now share their own experience of the disease, which translates into a form of knowledge that integrates with scientific and experiential knowledge. The MULTI-ACT project leverages both patient and other stakeholder experiences and increasing their ability to co-create and participate in decision-making processes in health research.
We produced the Patient Engagement Guidelines (Multi-Act Project, 2020) based on the lessons learnt from the landscape analysis of existing patient engagement procedures: literature review, web-search, interviews, surveys, and connections. They were developed into guidance, recommendations, methods and suggestions in line with existing good practice on guidelines production (WHO, 2014) and subsequently co-created with a series of actions including a public consultation, discussions, and reviews by the key stakeholders (experts, patients, researcher, clinical professional, policy makers, industries, etc.), and consolidated with two real life pilots made possible thanks to the collaboration established with existing multi-stakeholder research initiatives focusing on multiple sclerosis.
Master Scorecard
The Master Scorecard is a component of the CRIF which helps you implement co-accountability. It is a set of 125 indicators, from which your initiative will choose the most relevant ones for, creating a customised scorecard. The indicators used in the Master Scorecard come from an extensive literature review and from a co-creation process (especially for the patient-reported dimension).
The scorecard is intended for monitoring the initiative’s progress and assessing its impact. The selection is performed via the Collective Materiality Analysis, an auxiliary operative tool which allows you to engage all relevant stakeholders in your initiative in selecting the indicators. There are five dimensions of the Master Scorecard which reflect different areas of impact but also different and often competing interests of stakeholders involved in the research and innovation process as shown below.
· Efficacy: refers to the capacity of a given initiative or programme to achieve its mission (strategic priorities set via the stakeholder engagement process). For more, see Efficacy dimension.
· Excellence: concerns the quality of research and its findings. For more, see Excellence dimension.
· Social: considers the direct and indirect effects of health research for the whole society, going beyond patient needs. For more, see Social dimension.
· Economic: refers to long-term financial sustainability of health R&I initiatives. For more, see Economic dimension.
· Patient-reported: concerns patients whose needs and perspectives must be understood and incorporated into health research impact evaluation. For more, see Patient-reported dimension (PRD).
The dimensions are divided into 53 aspects, which are key topic areas.
The Master Scorecard translates the MULTI-ACT philosophy and your initiative’s agenda into action, providing indicators to evaluate the impact of health research and innovation on all stakeholders, with a special focus on the benefits for patients and society.
During the Master Scorecard’s development, we assessed a range of (health) research impact frameworks e.g., the Payback Model, the expected monetary value, the Research Impact Framework (RIF), the Research Excellence Framework (REF), logic models (Weiss, NIEHS), the Canadian Academy of Health Sciences model (CAHS), the research Impact Model (Kalucy et al., 2009; Graham et al., 2012; Ovseiko, Oancea and Buchan, 2012; Milat, Bauman and Redman, 2015; Raftery et al., 2016; Andreaus et al., 2019). Additionally, Social Return on Investment (SROI) (Jeremy Nicholls et al., 2012) was considered. These research frameworks offer different indicators to evaluate health research impact. However, they have some limitations concerning their suitability for assessing research from multi-stakeholder and multi-dimensional perspectives. First, they lack public (and specifically patient) engagement and multi-stakeholder participation in defining and selecting the indicators. Second, they provide a limited picture of multidimensional impacts as they focus on what is measurable rather than on relevant long-term social impacts.
The Master Scorecard is intended to be used as a strategic management tool for monitoring the progress of your initiative and for demonstrating how your initiative produces an actual social impact.
3.3 Digital Toolbox
The digital Toolbox is available at https://toolbox.multiact.eu. It is the web-based tool through which the CRIF is made available and thus an integral part of the MULTI-ACT project outcomes. Its components are described below.
In addition to the above, the Toolbox contains guidelines, instructions and additional materials, including the full text of this Manual and the Patient Engagement Guidelines. Using the Toolbox together with the CRIF Manual is the easiest way to familiarize yourself with CRIF and implement it. The Toolbox is intuitive, so you will not need any special guidelines to use it. The Toolbox is intended for continuous use: you can store documentation and stakeholder contacts there, update them, and re‑conduct Baseline Analysis, Collective Materiality Analysis and Patient Engagement Plan as needed.
3.4 CRIF Workflow
Following the evolution of the Co-accountability Pillars, a logic flow for implementation of CRIF has been defined. The CRIF Workflow will guide you through the adoption and implementation of the CRIF, emphasizing the crucial steps. It also shows how the Co-accountability Pillars and Governance Criteria work together. The CRIF Workflow described below shows the operative steps for your initiative to follow. The CRIF Workflow’s backbone is co-accountability; it enables the cyclical evolution of the agenda over time as a result of the initiative’s development or of external circumstances.
The Workflow’s 9 steps are clustered into 5 phases which directly correspond to the Co-accountability Pillars. The Workflow shows how the CRIF promotes continuous improvement. Furthermore, it embeds patient engagement in both the design of the most appropriate governance structure and bodies, the definition of the stakeholder engagement methodology and the definition of a tailored impact assessment system, thus enabling the concepts of “science with and of patient inputs” which is at the root of the MULTI-ACT patient engagement approach.
First, your initiative needs to define its scope and mission (phase 1), and then implement an operating framework which makes it possible to attain the mission (phase 2). It can control its results by defining specific impact aspects that matter most to the engaged stakeholders (phase 3) which are the basis for the selection of co-accountability indicators of a measurement model shared by the stakeholders involved in your initiative (phase 4). Finally, continuous monitoring of these indicators provides the basis for corrective actions (phase 5) to be taken in order to ensure that the agenda is aligned with the mission. For each of the phases described below, there are dedicated MULTI-ACT tools and corresponding Toolbox functionalities: Governance Criteria, Patient Engagement Guidelines, Collective Materiality Analysis and Master Scorecard. Being a flexible tool, the CRIF is not entirely chronological. However, some activities only make sense when performed before or after other activities. Below you will find a proposed sequence of activities. The Toolbox is designed in such a way that it will guide you and other users through the entire process.
If you do not find clear instructions at which stage to perform an action, it means you should act according to your best judgement.
Phase 1
The three steps of the phase 1 lead to the definition of your initiative’s mission. The mission usually remains unchanged in the long run:
· If your initiative is already set up, conduct a Baseline Analysis in order to measure its level of compliance with the Governance Model.
· Your initiative identifies its intended Beneficiaries, analyses its operating context, and learns about the needs of its stakeholders. If the “patients” stakeholder category is selected, then a patient engagement plan should be defined (see sub-criterion 2.1).
· On this basis, your initiative defines its new mission or refines an existing one.
Phase 2
Through the phase 2, MULTI-ACT proposes a specific methodology for defining the material topics which establish the agenda of the initiative: the Materiality Analysis. The materiality analysis is a way for your initiative to engage its stakeholders in defining which topics are significant and relevant for them. Based on that, the initiative can define next steps towards meeting their expectations.
Phase 3
Based on the material topics selected through the materiality analysis, your initiative can outline its agenda, identifying the transformative objectives that reflect the stakeholders’ perspective.
Phase 4
The agenda needs be monitored through a measurement system (relevant indicators associated with the material aspects are collected in the Master Scorecard). Once the indicators associated with the relevant aspects are identified, the initiative should put in place a consistent and efficient data collection procedure, in order to gather effectively and on a regular basis, the requested information.
Phase 5
At this stage, your initiative and its different stakeholders co-select aspects and indicators that best reflect their claims and interests. You are strongly encouraged to use the dedicated functionality in the Toolbox for this process. Your initiative’s own score card should contain 12-15 aspects chosen from a list of 53, and 12-15 indicators chosen among the 125 that the model makes available in its impact assessment scorecard. The circle closes with the publication of the periodic report of the initiative, which MULTI-ACT suggests to produce annually and which provides the basis for the analysis of the differences between what was planned and what was achieved, allowing to identify the appropriate improvements of the agenda of the initiative. While the mission is defined at the beginning of the initiative, the alignment of the agenda with the mission needs to be monitored and checked regularly, and therefore, phases 2 to 5 should be repeated accordingly (e.g. on an annual basis). Your initiative needs to base the entire process (phases 1 to 5) and application of the tools on continuous engagement with its stakeholders, especially patients. Patient Engagement Guidelines will help you do it correctly.
4 GOVERNANCE
As you can see in the, the Governance Criteria are constructed in a hierarchical manner. They set the main areas of governance. Each criterion is divided into several sub-areas: sub-criteria. The order is thematic, not chronological, so they are more like tasks to be accomplished that steps to follow. In each sub-criterion, there is at least one recommendation. Recommendations concisely describe the actions that an initiative needs to take to achieve the goal outlined in the sub-criterion. They are accompanied by detailed explanations of the actions and concepts behind them.
Many of the recommendations give instructions on how to structure governance bodies, how to set their areas of responsibility and rule of participation in them. For your convenience, there is a Governance bodies section which summarizes information about the governance bodies in one place.
First, check your initiative’s compliance with the Governance Model (both the Criteria and the Patient Engagement) through the Baseline Analysis, and then focus on the areas identified as gaps. Your initiative then may focus on implementing these specific recommendations in order to become compliant with the Model. Below you will find the full text of the five Governance Criteria. We encourage you to read them in full at least once, so you will have an overall understanding of all the key concepts and how they relate each other. To make the implementation of the recommendations easier, the Governance Model’s flexibility leaves your initiative as much discretion as possible, so that you can implement the recommendations in the most suitable way for your specific circumstances and mission.
Since the Criteria deal with stakeholder engagement and governance structures and procedures, it is worth understanding CRIF’s stakeholder typology (presented below) and its approach to the governance structure before starting.
4.1 Stakeholder typology
Stakeholder is an individual or group that is affected by the outcomes of your initiatives’ actions, or who can influence these outcomes or may have an interest in them(Freeman, 1984). In other words, stakeholders are people, communities, organisations and other entities that experience a change – positive, negative or neither – as a result of the activities of your initiative. Some of them will be participants of your initiative, others will not be even aware of its existence. Using a stakeholder classification is a useful way of thinking about people and organizations relevant to your initiative. While not all of them are equally relevant for your research, nor are they all going to be involved to the same degree, it is important not to overlook any group influenced by your initiative.
Additionally, the CRIF Manual refers to Promoters, Appliers and Beneficiaries of the Collective Research Impact Framework, when describing how stakeholder organizations and their representatives participate in governance of your initiative.
4.2 Governance bodies
Governance bodies are groups with specific roles within a multi-stakeholder initiative that are composed of individuals participating to the initiative itself. In CRIF, it is crucial to ensure both participation and balance of power of different stakeholder categories in the bodies. The suggested governance bodies to be established are presented in the figure below.
Information about functions, composition and significance of the governance bodies are described in the Governance Criteria. In the tables below, you can find condensed summaries of the functions, composition and appointment procedures for each governance body, with references to the Governance Criteria. It may prove useful later on when you decide to set up a governance body or compare characteristics of the bodies that already exist in your initiative with those set out by the CRIF.
Leadership Board (LB)
|
Function |
LB is the decision-making body within the governance structure (recommendation 2.3.1). It oversees fulfilment of the mission and agenda, and coordination and implementation of the activities of your initiative. It supervises Working Groups (WGs), Committees, and administration (recommendation 2.3.2). It enforces deadlines and improves your initiative’s performance (sub-criterion 4.2), with help from the Management Team, if needed. The LB evaluates and chooses actions and tools (e.g. Progress Report, to respond to current needs of the Beneficiaries and changing circumstances) (sub-criterion 4.3). It delegates tasks to WGs or other bodies, as needed. It leads the review process (recommendation 5.1.7), with the Stakeholder Advisory Board (SAB). LB creates a procedure formalizing various aspects of how the initiative functions, from governance bodies appointment to stakeholder engagement (recommendation 2.3.3), with support of Compliance Committee and the Engagement Coordination Team. This procedure needs to be approved by the SAB. LB appoints (recommendation 2.3.2): · Chair/coordinator acting as an internal and external point of reference for the initiative; · Operational teams, such as a sub-board and the Secretariat/ Management Team – when needed; · Working Groups, Committees, and collaborative team to carry out various tasks LB is responsible for formalizing procedures and strategies: · It creates the Engagement Plan (sub-criterion 3.1, Implement phase) with ECT and Stakeholder Advisory Board (SAB). · LB has a responsibility to define the collective Action Plan and enforce its implementation (sub-criterion 4.1) through establishment of dedicated WGs, and creating accountability mechanisms. · It creates a process for collecting feedback, opinions, and grievances of internal and external stakeholders (recommendation 5.2.1), with the ECT. · It formalizes procedures on how the initiative’s participants interact with each other, balancing stakeholder engagement and agile management (sub-criterion 3.4) with the ECT and the CC. The LB also determines the budget and conduct a cost analysis of the initiative, as well as identifies critical issues and gaps in your initiative’s operations (sub-criterion 4.4). It may delegate these tasks to the Secretariat/Management Team. LB Identifies gaps in stakeholder engagement capacity (sub-criterion 3.1) and then monitors, evaluates and improves quality of stakeholder engagement (sub-criterion 3.1), with SAB. It ensures appropriate communication to relevant stakeholders and the general public (recommendation 5.3.1) The LB is responsible for constantly maintaining an alignment between the shared assessment system and the mission and agenda of the initiative (recommendations 5.1.4 and 5.1.5). |
|
Appointment |
LB is set up by the Promoters. Composition of the LB needs to be approved by the SAB and PAB (sub-criterion 2.3). |
|
Composition |
The composition of the LB reflects the categories of the stakeholders that participate in your initiative and have strategic importance. Its members act as these categories’ representatives. Their number varies according to the initiative’s needs. LB has to be balanced in terms of gender, sector and geographical background, language, political diversity, perspectives and experiences. The members of LB should be committed and skilled individuals, which should ensure constant participation to the initiative’s development. LB members hold equal power because it guarantees equity among participant stakeholders. The composition of the LB and its members should undergo the approval of the SAB and the PAB (recommendation 2.3.2). |
Engagement Coordination Team (ECT)
|
Function |
The ECT coordinates the involvement of stakeholders, including patients, in all the operations. It coordinates all training and coaching activities to facilitate the stakeholders’ engagement (sub-criterion 3.2), which includes providing briefing materials and organizing training sessions. Cooperation between the ECT and the Leadership Board (LB) plays a crucial role in the initiative’s governance. While the LB provides agile management, the ECT should guarantee and facilitate the participation of weak and/or marginalized stakeholders as well as a balance among different points of view (sub-criterion 3.4). · The ECT works as a facilitator between the Stakeholder Advisory Board (SAB) and the LB. (recommendation 2.1.1). · It identifies gaps in stakeholder engagement capacity (sub-criterion 3.1), together with the LB. · It assesses stakeholder engagement (sub-criterion 3.1), together with the LB and the SAB. · The ECT maintains the active participation of the internal stakeholders in the LB-led process of setting up a stakeholder feedback mechanism (sub-criterion 5.2). In terms of patient engagement responsibilities, the ECT: (full description in the Composition and skills of Engagement Coordination Team): · Designs, implements and monitors the Patient Engagement Plan. · Makes sure that the experiential knowledge of the patients is used to improve patient-reported outcomes. · Moderates interdisciplinary dialogue. · Translates technical language into a language that patients easily understand · Mitigates issues like ethical conflicts in protocol design, tokenism, patient recruitment etc. |
|
Appointment |
Promoters establish the ECT: they recruit and appoint its members. The agreement of the LB is needed (sub-criterion 2.3, recommendation 2.1.1). You can find detailed instruction Establish the Engagement Coordination Team. |
|
Composition |
The composition of the ECT is described in detail in the Patient Engagement Guidelines. However, the composition of this team can vary depending on the specificity of individual programs and projects. All recruited experts are encouraged to undertake additional training. |
Stakeholder Advisory Board (SAB)
|
Function |
The main function of the SAB is advisory – it supports the Leadership Board (LB). It may, however, evolve over time into a decision-making body, acting like a Stakeholder Assembly. The SAB leads the review process (recommendation 5.1.7) with the LB. It confirms appointment of the Compliance Committee (CC), with the LB (sub-criterion 2.4). Patients, as a specific stakeholder category included in the SAB, may be asked by the LB for their own contribution. This group may form a sub-board of SAB: the Patient Advisory Board (PAB) (recommendations 2.1.1, 2.3.1). It approves the composition of the LB, with the PAB (sub-criterion 2.3). |
|
Appointment |
Appointed by Promoters with the contribution of the Compliance Committee (CC) and the Engagement Coordination Team (ECT) (recommendation 2.3.1). |
|
Composition |
The SAB is composed of interested stakeholders. The Promoters, with the ECT, arrange an open call for participation in the SAB. The CC and the ECT establish the rules regarding selection, composition, and balance of the SAB. PAB is a sub-board of SAB (recommendation 2.3.1). |
Patient Advisory Board (PAB)
|
Function |
PAB may be a separate body or group representing patients within the Stakeholder Advisory Board (SAB). It presents the voice and opinions of patients, including underrepresented patients (recommendation 2.1.1). It is to be consulted by the Engagement Coordination Team (ECT) and the Leadership Board (LB). It approves the composition of the LB, with the SAB (sub-criterion 2.3). |
|
Appointment |
Promoters with the Compliance Committee (CC) and ECT appoint PAB during creation of the SAB (sub-criterion 2.3). |
|
Composition |
PAB is composed of patient representative from the SAB (recommendation 2.1.1). |
Compliance Committee (CC)
|
Function |
CC maintains a balance among stakeholders’ stances and expectations. It oversees ethical issues too (sub-criterion 2.4). The CC takes part in the decision-making process of your initiative. It contributes to the Leadership Board’s (LB) activities, especially: · Guaranteeing equity (sub-criterion 2.4) · Ensuring that the self-interest of stakeholders does not prevail on collective decision-making processes (sub-criterion 2.4) · avoiding tokenism (sub-criterion 2.4) · Making sure that the decision-making process considers different views (sub-criterion 2.4) · Managing conflicts (sub-criterion 2.4) · Guaranteeing ethical acceptability and social justice of the initiatives’ objectives and activities (sub-criterion 1.4); · Ensuring a balance between effective engagement of participants and agile management of the initiative (sub-criterion 3.4); · Supporting the LB in formalizing a procedure (recommendation 2.3.3). It also may support Secretariat/Management Team if needed in its duties related to financial oversight (sub-criterion 4.4). |
|
Appointment |
First appointed by the Promoters in the beginning of Governance Model implementation, later officially confirmed by the LB and the Stakeholder Advisory Board (SAB) (sub-criterion 2.4). |
|
Composition |
It can be a committee or an individual, depending on the size, level of development and resources of your initiative (sub-criterion 2.4). |
Committees and Working Groups (WGs)
|
Function |
Creation of WGs is optional, and their responsibilities, role and specific tasks are assigned according to current needs of your initiative. For example, the Leadership Board (LB) may charge the bodies with research or reporting. They may also be responsible for maintaining feedback mechanism and communication (recommendation 5.3.1). WGs may carry out operative tasks, while Committees may provide insights and opinions (recommendation 2.3.1). They report to the LB and are supervised by the Management Team. |
|
Appointment |
LB appoints WGs if needed and as needed (sub-criterion 2.3). |
|
Composition |
The WGs should be composed and balanced in terms of the stakeholders’ categories and needs of the initiative (recommendation 4.1.1). |
Secretariat/Management Team
|
Funtion |
The Secretariat and Management Team may be two different bodies or one. It depends on the size and structure of the multi-stakeholder initiative. Secretariat/Management Team supervises administrative and operational tasks. The body: · Enforces the deadlines and oversees activities in your initiative (sub-criterion 4.2). · Supervises of the general performance of the initiative according to the defined mission and agenda (sub-criterion 4.2). · Helps the Leadership Board (LB) with the Progress Report (recommendation 5.1.6). · Runs administrative duties. · Ensures of financial security and legal compliance of your initiative (sub-criterion 4.4). · Oversees Working Group’s (WG) activities. · Supports the LB in gathering data for Progress Reports (recommendation 5.1.6). · May support the LB and the Stakeholder Advisory Board (SAB) in the review process (recommendation 5.1.7). · Supports the LB in setting up a process for gathering stakeholders’ feedback (recommendation 5.2.1). The LB may decide to delegate to the Secretariat/Management Team the tasks of determining the budget and conducting a cost analysis of the initiative, as well as identifying critical issues and gaps in your initiative’s operations (sub-criterion 4.4). |
|
Appointment |
The LB appoints it/them based on the initiative’s needs and tasks to be performed (sub-criterion 4.2). |
|
Composition |
The LB can decide on the composition of these bodies (or body). A multi-stakeholder approach is not required here. |
4.3 Governance Criteria
In this section, you will find the full text of the Governance Criteria. You can see the structure of the Governance Criteria in the table below.
Criterion 1: Mission and agenda
In the process of formulating a mission and a shared agenda for your initiative, it is important that Appliers:
• Identify the initiative’s intended Beneficiaries and analyse the context in which it operates;
• Define a shared mission and common agenda;
• Promote a movement-building approach to achieve transformative changes;
• Guarantee ethical acceptability and social justice.
Sub-criterion 1.1: Identify intended Beneficiaries, analyse the operating context of the initiative and understand the needs of stakeholders
Recommendation 1.1.1: Be aware of who are the initiative’s intended Beneficiaries and have clear strategies to facilitate their active participation
Recommendation 1.1.2: Carry out a context analysis to understand the operating context of the initiative and identify the needs of its stakeholders, with particular regard to the intended Beneficiaries
The Appliers identify the intended Beneficiaries and set clear strategies to engage them and enable their participation (in this regard, please refer to sub-criterion 2.1 and 3.2). In the health R&I, society and patients are the key Beneficiaries, and the ultimate goal is to improve their health and well‑being. The Appliers should explicitly identify these Beneficiaries, their characteristics, and their needs. This step is necessary for the identification of the initiative’s long-term goals later on.
The initiative also conducts a context analysis. Its purpose is to identify the main actors and trends that may be challenging for the initiative, as well as risks and assumptions that may affect its performance. Context analysis involves looking at the current state of the “issue” that your initiative seeks to influence or the problem it seeks to solve: its social, environmental, and political conditions, actors who may be able to bring change. This is why, before defining the mission and agenda (see sub‑criterion 1.2), Appliers first analyse which “ecosystems” and communities are affected, what key issues and pressures are faced, and the main social, political, economic, and technological factors that together create the context.
It is recommended to carry out the context analysis in parallel with the Plan phase of sub-criterion 3.1, which describes profiling and mapping of the stakeholders.
Having identified the intended Beneficiaries, analysed its operating context, and mapped its stakeholders, your initiative is ready to deepen its understanding of the stakeholders’ needs. Needs assessment is a fundamental process that leads to a better understanding of the challenges faced by the initiative and its stakeholders. You can use it to identify the change that your initiative wants to bring about in society. This change will be subsequently expressed through the initiative’s mission and detailed through its agenda, as described in sub-criterion 1.2.
The need assessment is also related to sub-criterion 2.2, which recommends initiatives to set up an initial consultation process to understand the bottom-up needs and challenges of the potential participants of the initiative.
It is possible to integrate the context analysis and needs assessment: Appliers can identify the problem faced, its main roots, and its most relevant consequences, involving relevant stakeholders in this analysis. In the process, the stakeholders present their needs.
This exercise may facilitate the steps described in the following sub-criterion 1.2, namely the definition of the mission and the agenda.
Sub-criterion 1.2: Define a shared mission and common agenda
Recommendation 1.2.1: Define a shared mission and a common agenda involving relevant stakeholders, thus tackling the intended issue with a unifying long-term vision and a clearly defined set of objectives and actions necessary to pursue the mission.
Recommendation 1.2.2: Identify appropriate indicators in alignment with the initiative relevant aspects and objectives considering the different perspectives of the stakeholders involved.
Initiatives adopting the CRIF have in common the vision of striving to conduct mission-oriented research. They define their mission and agenda according to their specific vision and unique circumstances.
Mission definition
A mission statement defines your initiative’s current and future role, its goals[1], and its approach to reaching them.The mission statement includes:
· Descriptive elements clearly illustrating what the initiative wants to achieve;
· Transformative elements i.e. the changes the initiative wants to create in the context in which it operates.
With regard to the descriptive elements, your initiative may want to describe:
· Its potential Beneficiaries;
· The scope of the intervention (e.g. health domain, geographical area, gender, socio-economic conditions).
With regard to the transformative elements, your initiative may need to clarify:
· The expected change intended to happen for the Beneficiary;
· A baseline against which this change could be assessed.
Example of a research initiative mission (Mazzucato, 2018) |
|
Decreasing the burden of dementia by 2030 reducing the progression of the disease in affected patients in Europe. Descriptive elements: 1. Beneficiary: affected patients 2. Scope of the intervention: dementia brain disease in Europe Transformative elements: 3. Expected change for the Beneficiary: reducing the progression of the disease 4. Baseline: the current burden of dementia. |
Materiality analysis and identification of aspects
According to the criterion 5, Appliers need to establish a shared and effective assessment system, and a mechanism to receive feedback. The assessment system must include a set of indicators and promote continuous improvement, and communication.
Then, the Appliers enable the stakeholders to co-select measurable objectives in order to assess the progress and outcomes of the initiative. The initiative’s governance bodies, on other hand, identify the aspects of measurement through a process that requires identification of measurable and achievable targets: in this way, they ensure stakeholders’ engagement over time.
In order to assure coherence between the indicators used in monitoring and reporting and the interests of different stakeholder categories involved, the initiative carries out a materiality analysis.
Materiality analysis is a managerial tool that can facilitate the adoption of co-accountability and multidimensional impact assessment (Master Scorecard). It allows you to gather stakeholders’ perspectives and to identify the CRIF aspects that are significant for stakeholders. From this point of view, materiality analysis can be defined as a bridge between your initiative’s mission and the outcomes of the research it conducts. It links the reasons why the initiative was established with the results that matter most to the stakeholders. You will find detailed instructions on how to conduct it in the Materiality Analysis section.
Agenda definition
An agenda is a list of fundamental transformative objectives (i.e. priorities), including a description of the main outputs[2] and activities needed to achieve them. It is agreed upon by stakeholders and your initiative will aim to achieve its agenda in order to fulfil its mission. The agenda must be consistent with the aspects selected as relevant during the materiality analysis. For each priority of the agenda, the initiative formulates:
• A transformative objective, describing the type of intervention and the transformative threshold and baseline according to which the initiative considers its intervention successful
• Outputs and related activities needed to reach the transformative objective
Once they are defined, your initiative ensures proper dissemination and circulation among all involved stakeholders of the agenda, timeline, and objective that should be shared among all team members.
|
Example of a research initiative (Mazzucato, 2018) |
|
Agenda of Dementia Care Initiative (timeline 2020-2030) · Increase the percentage of dementia patients who are given personalized treatments, through the development of a customizable therapy protocol, according to specific patients’ needs, to be shared with an “X” number of medical facilities. · Increase the dementia patients’ feelings of being more physically and intellectually independent through the development of a customised, free smartphone and computer IT application to be easily accessed by patients in Europe to perform daily tasks. · Increase the percentage of early-diagnosed (within one year from the disease start) dementia patients in Europe through development of a digital application (e.g. background app linked to smartphone and computer) that is able to detect early symptoms of neurodegenerative diseases and recommend prompt treatment to users, to be available on at least on 2 operative systems (e.g. Android and IOS). The transformative objective (priority): The number of dementia patients who are given personalized treatments in Europe is increased by * %. Outputs: Development and adoption of a customizable therapy protocol according to specific patients’ needs. Activities: Research and development of the customizable therapy protocol. Assumption considered: • If patients could get personalized treatment, the progression of the disease could be slowed down up to * %. • If patients would feel more independent in performing daily tasks, the feeling of the disease burden could be decreased. Furthermore, performing these tasks could also be a stimulating activity to slow down the progression of the disease. • If dementia patients are diagnosed earlier, the burden of the disease is drastically decreased thanks to the specific therapies patients can adopt. |
When defining the agenda, always keep in mind the relevant aspects in order to ensure the alignment between the assessment system and the mission, agenda and objectives of the initiative. In the case of initiatives at an advanced stage of development, which have already defined and tested a mature governance model, an additional internal control system can be introduced in order to measure progress towards its agenda and the achievement of its transformative objectives. In this regard, the box below gives some further suggestions[3].
|
Timeline and Coherence Check |
|
Once defined the agenda, the initiative should monitor the timeline of the intervention, namely the temporal and operation feasibility needed to achieve the objective. For instance, your initiative could answer the following questions: 1) In what timeframe is it reasonable to reach our objective? 2) Is it in line with our initiative’s timeframe? 3) Is the threshold identified as our expected results realistically achievable? Can we contextualize the number? Did we make explicit the reference I am using to set up my percentage for my objective? Having defined the priorities of the agenda, the initiative should ensure the coherence and the causal link among activities, outputs, objectives, agenda and mission. In this regard the activity should lead to the output, the output – completely under the responsibility of the project – should lead to the objective. For instance, the initiative could answer the following questions: · Is the agenda contributing to the mission statement? In which way? · Are we accountable 100% over the activity and outputs? Are the activities contributing to the agenda? In which way? |
Finally, ensure secure funding to guarantee adequate resources for the development and the correct deployment of activities, as defined in sub-criterion 4.4. In particular, implement an effective cost‑management process, i.e. focus on the determination of the needed budget, cost analysis, and identification of gaps and critical issues.
The table below offers a set of additional data to be considered when defining the mission and agenda. In this last regard, please consider that the expected impact could be influenced by several factors both in and out of control of your initiative.
|
MULTI-ACT definition |
Description |
Question to answer |
Timing |
Sphere of control / influence |
|
Mission |
The initiative’s current and future role, its goals and its approach to reach them |
What is the long-term goal of the initiative? |
Long term |
Influence |
|
Agenda |
The transformative objective, describing the type of intervention and the transformative threshold and baseline (according to which the initiative considers its intervention successful) |
Why/What do I want to achieve? Which change do I want to contribute to/to bring about? |
Medium- to long term |
Influence |
|
The outputs needed to reach the transformative objective |
How do I want to achieve it? Which concrete actions do I need to put in place? |
Short term |
Control |
|
|
Activity |
Which activities will I perform? |
Short term |
Control |
Sub-criterion 1.3: Promote a movement building approach to achieve transformative changes
Recommendation 1.3.1: Promote a movement building approach throughout all the initiative phases by enabling the generation of a community aspiration, becoming a platform that fosters change and innovation, engaging stakeholders in long term strategic action, enacting constant learning mechanisms and enabling authentic involvement of community
Recommendation 1.3.2 Be transformative and disruptive by promoting innovative problem-solving and critical thinking approach among involved stakeholders, in order to open new horizons for the research and go beyond the boundaries of the current research system, with the aim of achieving collective social impact
Appliers of the CRIF should embody a movement-building approach (Cabaj and Weaver, 2016) by integrating the above recommendations. In order to promote a movement-building approach and achieve transformative changes, your initiative:
· Creates a sense of aspiration shared by the stakeholders in which everyone agrees and works together toward the achievement of the related goals;
· Tries creating a “container for change” that seeks the change of the people involved in your initiative;
· Engages in long-term (strategic) actions, at all stages of the project;
· Focuses efforts on activities that result in a greater opportunity for change. This is achieved by having the agents participate and collaborate in long-term or strategic actions;
· Incorporates a shared measurement process as part of a complete sharing learning process in which participant members “hold each other accountable and learn from each other’s successes and failures” (Kania and Kramer, 2011). In this sense, the shared impact assessment serves as a resource to provide feedback to the system and serve as a constant learning mechanism;
· Ensures authentic community engagement including those negatively affected by certain measures in the process of change.
Sub-criterion 1.4: Guarantee ethical acceptability and social justice
Recommendation 1.4.1: Consider societal relevance and ethical acceptability of the initiative while minimizing potential unintended negative consequences
Recommendation 1.4.2: Aim to extend the positive impact of research to as many people as possible and ensure social justice
Appliers consider how relevant their initiative’s objectives are for the society and how to maximize its positive impacts while minimizing its negative consequences and ensuring that the rules of social justice are reinforced.
This recommendation is of qualitative nature and should be considered as a guidance and a reference to be applied throughout the entire process of decision-making. The responsibility of ensuring the consideration of this recommendation throughout the entire process could be assigned to the Compliance Committee, a body described in detail within sub-criterion 2.4.
Criterion 2: Participatory Governance
Appliers should guarantee an inclusive and equitable governance model promoting the involvement of all interested parties through a co-designing approach. To this end, ensure that the initiative:
· Allows the involvement of private intended Beneficiaries;
· Adopts a multi-stakeholder approach enabling co-creation;
· Implements a participatory structure;
· Guarantees equity and mechanisms to avoid self-interest.
Sub-criterion 2.1: Allow the involvement of intended Beneficiaries
Recommendation 2.1.1: Involve intended Beneficiaries in the agenda design, in the decision-making process and in the initiative development, implementation and assessment. For the purpose of MULTI-ACT, patients are usually the intended Beneficiaries. With specific regard to patients, develop a roadmap to capture “experiential knowledge” of patients, to better understand how to draw on their experience and use the experience constructively for co‐creation purposes and to evaluate the impact of research on the outcomes that matter to patients.
MULTI-ACT proposes a set of guidelines to support the engagement of patients which aim at leveraging patients together with the other stakeholders’ experience and at raising their ability to co‐create and participate to decision‐making processes.
The involvement of patients – defined as the intended Beneficiaries – is pivotal in the implementation of the CRIF. In this regard, MULTI-ACT proposes a path for patient engagement to ensure that people affected by brain diseases are given an equal voice with other stakeholders. To ensure continuous engagement of patients throughout the entire initiative and give them authentic influence, Appliers:
1) Appoint an Engagement Coordination Team (ECT) that will be in charge of coordinating the involvement of stakeholders, including patients, in all the operations. Initially, you (the Promotes) appoint the ECT, and the LB later accepts your choice or suggests a different composition.
2) Create a Patient Advisory Board (PAB), a specific group of patients within the Stakeholders Advisory Board (SAB), to be involved and engaged throughout the entire development of the initiative, providing advice, insight, and perspectives on the initiative’s activities and operations.
For the details about roles, responsibilities, appointment procedures, and structure of the above bodies, please refer to the Governance Bodies section. ECT is additionally discussed in a section of the Roadmap Action 1: Establishment of an Engagement Coordination Team (ECT).
Sub-criterion 2.2: Adopt a multi-stakeholder approach enabling co-creation
Recommendation 2.2.1: Prepare the initiative to implement co-creation processes by framing/reframing the composition of the initiative according to the new multi-stakeholder nature
Recommendation 2.2.2: Set up an initial consultation process in order to understand the bottom-up needs and challenges of the potential participants of the initiative
Multi-stakeholder approach to governance is essential for co-creation to happen. Co-creation may be defined as co-operation and learning from one other to raise awareness on important issues and to build relationships between groups and individuals (Cottam and Leadbeater, 2004), with particular attention to those that normally do not interact. In order to adopt the multi-stakeholder approach, your initiative needs to build participatory governance structures and processes, which are designed to create shared ownership of among its stakeholders (i.e. you – the Promoter, patients, care providers – medical professionals, the industry, research institutions etc.). To shape the governance structure of your initiative that would be compliant with the multi-stakeholder perspective, first you need to identify the structure and tools best suited to help your initiative achieve its objectives.
To achieve this goal, the initiative first analyses its current composition and envisions a stakeholder structure that would be ideal for achieving its mission and agenda. This activity allows to map the potential gaps in terms of stakeholder composition and to ensure that the initiative involves participants from all the relevant stakeholder categories. Once the initiative defines its composition, it identifies and considers stakeholders’ needs, challenges, and barriers to guarantee genuine participation.
In order to accomplish this goal, conduct the analysis described below:
1) Analyse the current structure of the initiative, its organizational model, and its current participant composition (if your initiative already exists). Envision, what would be ideal for achieving your mission and agenda.
2) Identify the stakeholders’ categories that could be involved according to the context and the objectives pursued by the initiative and, therefore, that could be potential participants in the initiative.
3) Identify the potential relevant gaps in terms of stakeholder composition and, if applicable, integrate the participation of those stakeholder categories that are missing according to the above-mentioned point 2; Ensure that your initiative involves participants from all the relevant stakeholder categories.
4) Identify and consider stakeholders’ main needs, challenges, and barriers to guarantee their genuine and committed participation.
Conducting this analysis is your task as a Promoter. It precedes structuring of the governance model of the initiative itself, the composition of its bodies, and the formalization of the structure, participants, and roles, which will be explained in the Sub-criterion 2.3: Implement a participatory structure and in the Governance Bodies section.
You will integrate the results of this analysis with the activities described in the Sub-criterion 3.1: Define and approve a methodology to engage stakeholders.
Sub-criterion 2.3: Implement a participatory structure
Recommendation 2.3.1: Define a clear and agile backbone structure and define clear roles and responsibilities of all involved stakeholders, based on the mission and the agenda
The participatory structure is the system by which an organization makes and implements decisions in pursuit of its strategic objectives. Appliers will need to adapt the structure to the organizational model proposed below or, if they are new-born organizations, define their structures accordingly. The section Governance bodies describes in detail the main bodies of the MULTI-ACT Governance Model: their main functions within the structure, process of appointment, and stakeholder composition. It is crucial that you become familiar with the content of this chapter.
The roles of the other bodies are further described under specific sub-criteria and in the Governance bodies section. In particular, the Leadership Board is described in the recommendation 2.3.2, the Working Groups in recommendation 4.1.1, the Engagement Coordination Team (ECT) in sub-criterion 2.1, and the Compliance Committee in sub-criterion 2.4. You, with assistance from the ECT, are responsible for arranging an open call to interested stakeholders for participation in the Stakeholder Advisory Board (SAB). Establish rules for selection, composition, and balance of the SAB with the contribution of the Compliance Committee and the ECT.
Recommendation 2.3.2: Identify a mix of committed and skilled individuals that will be a part of the Leadership Board and balance them in terms of gender, sector background, geographical background, language, political diversity, opinion and experience
Set up the Leadership Board (LB), comprising of at least one representative from each category of stakeholder (categories of stakeholders are defined in the recommendation 2.2.2 and sub-criterion 3.3). The composition of the LB should be balanced in terms of gender, sector and geographical background, language, political diversity, perspectives, and experiences. The members of LB should be committed and skilled individuals, which should ensure constant participation in the initiative’s development. The members of the LB should have equal power, in order to guarantee equity among participant stakeholders. The composition of the LB and its members should undergo the approval of the SAB and the PAB.
Specific activities, roles, and responsibilities of the LB are described and formalized within a procedure as pointed out in the recommendation 2.3.3 and in the governance bodies section. The LB appoints a chair/coordinator who will become the internal and external point of reference to the initiative. It may also create an operational team, such as a sub-board (the executive team) and a secretary (supporting operations).
Recommendation 2.3.3: Formalize how the stakeholders involved in the governance will interact with each other and cooperate within the governance structure
As an initiative, adopt a formal procedure, which will be public and will transparently define:
· Which is the governance structure of your initiative,
· How the governance bodies are composed,
· How members are appointed,
· How decision-making processes are handled,
· How stakeholders and the public might participate in the initiative and/or take part in its governance bodies or in other bodies.
An example of how the procedure could be structured is reported below:
· roles and responsibilities,
· structure and membership of the governance bodies,
· operations (i.e. regular operations and meetings),
· relations between the governance bodies,
· External relations and public involvement.
The Leadership Board (LB) has the responsibility of developing this procedure, with the support and contribution of the Compliance Committee (CC) and the Engagement Coordination Team (ECT). The resulting document should be shared and approved by the Stakeholder Advisory Board (SAB).
Sub-criterion 2.4: Guarantee equity and mechanisms to avoid self-interest
Recommendation 2.4.1: Guarantee the support to and the meaningful participation of disadvantaged stakeholders (for financial, communication, language, cultural, age or mobility reasons) through appropriate mechanisms to give voice to each of them and avoid marginalization
Recommendation 2.4.2: Ensure that monitoring measures are put in place to protect the integrity and multi-stakeholder nature of the initiative and manage potential conflicts, considering that different views have to be accommodated in the decision-making process
Recommendation 2.4.3: Implement appropriate engagement mechanisms to create and maintain commitment and ownership among the participating stakeholders
To guarantee equity and implement mechanisms preventing self-interested actions of stakeholders, a specialized body is needed. To this end, you appoint the Compliance Committee (CC); this decision needs to be later confirmed by the Leadership Board (LB) and the Stakeholder Advisory Board (SAB). It will be in charge of maintaining a balance among stakeholders’ influences and expectations and overseeing the ethical issues that may arise during the implementation of the initiative. More on the composition, appointment, and functions of the CC in the Governance bodies section.
The CC represents the point of reference for the implementation of recommendations 2.4.1, 2.4.2, 2.4.3, and with regard to those included in sub-criterion 1.4 and sub-criterion 3.4.
Criterion 3: Clear, effective and inclusive methodology of stakeholder engagement
Appliers of the CRIF are able to guarantee a comprehensive, balanced, and efficient stakeholder engagement process, ensuring the participation of patients and caregivers, and of other relevant stakeholders, by:
· Defining and approve a stakeholder engagement methodology;
· Engaging private intended Beneficiaries;
· Differentiating the level of engagement according to participants;
· Ensuring a balance between the engagement of participants and agile management of the initiative.
Since CRIF is a collaborative tool, which requires the involvement of stakeholders in the entire governance process, this criterion works as an overarching principle for the other four governance criteria. In addition, Patient Engagement Guidelines provide a methodology to engage the stakeholder “patient” and facilitate development of a roadmap to capture patients’ voices and help them to co-create with the other stakeholders’ experience.
Sub-criterion 3.1: Define and approve a methodology to engage stakeholders
Recommendation 3.1.1: Define a methodology to engage stakeholders, create and maintain an open dialogue with them and manage the engagement processes of participants throughout the entire design and implementation of the health research initiative
Recommendation 3.1.2: Provide clear information regarding why the initiative is engaging (the purpose), what issues to engage on (the scope), and who needs to be involved in the engagement
This fundamental process relates to the engagement of stakeholders who cooperate towards the achievement of the objectives of the initiative. The Appliers define and implement a structured and detailed methodology to effectively engage those stakeholders who are of strategic importance, so they can cooperate towards the achievement of the objectives of the initiative.
Successful engagement depends on deep understanding why an organization is engaging (the purpose), what issues to engage on (the scope), and who needs to be involved in the engagement (the stakeholders). An engagement process should clearly describe:
· How to establish commitment;
· How to determine the purpose, scope, and stakeholders of the engagement;
· How to integrate stakeholder engagement within the governance;
· How to carry out the processes that will deliver quality and inclusive engagement practices, and valuable outcomes.
The methodology of stakeholder engagement should comprise at least some key phases, which can be summarized as follows:
a) Plan – identify which stakeholders should be engaged in your initiative due to their strategic importance to achieve your mission. Cluster them into different categories that reflect different levels of engagement. Determine the rights, duties, and responsibilities for each category of stakeholders.
b) Prepare – when you identify the stakeholders and determined the levels of engagement, assess:
i) the different characteristics and needs that these stakeholders may have;
ii) barriers concerning their effective engagement;
iii) risks related to the involvement of such a diverse group of actors.
c) Implement – define activities that will allow the participation of stakeholders in your initiative through formalized procedures that define in detail the interaction and cooperation between the different actors.
d) Review and improve – put in place mechanisms that would guarantee the monitoring and evaluation of the stakeholders’ engagement in order to improve it.
In the Plan phase:
· Profile and map your stakeholders: To design the stakeholder engagement process, you need a clear understanding of who the relevant stakeholders are, and how and why they may want to engage with your initiative. Profiling and mapping shall be reviewed and revised throughout the process and for this reason, it should be formalized. It is recommended to carry out the stakeholder profiling analysis in parallel with the context analysis as described in sub-criterion 1.1.
· Determine their levels of engagement: Map and cluster stakeholders into different categories to determine which groups and individuals are most important to be engaged from the point of view of the engagement process’s purpose and scope (please refer also to sub-criterion 3.4). Define different levels of engagement, which determine the different rights, duties, and responsibilities of the interested stakeholders. Defined levels are also used to establish the composition of the SAB (please also refer to sub-criterion 2.3).
In the Prepare phase:
· Build capacity: Different actors have different levels of expertise, confidence, and experience. Some individuals and groups may find it difficult to take up your invitation to engage, or their circumstances may hinder them from fully contributing to the process. This may be due to language, literacy, disability, or cultural barriers, problems of geographical distance, or lack of time, or gaps in their knowledge about a specific issue. The Leadership Board (LB), with the help of the Engagement Coordination Team (ECT), should timely identify where engagement capacity needs to be built , in order to avoid exclusion of these stakeholders, or to prevent them from disengaging (please also refer to the sub-criterion 3.2).
· Identify and prepare for engagement risks: In order to formally identify, assess, and address engagement risks, Promoters you need to perform a risk assessment. The potential stakeholder risks could be, for instance: unwillingness to engage, participation fatigue, creating expectations of change that the organization is unwilling or unable to fulfil, a conflict between participating stakeholders, etc.
· Invite and properly brief stakeholders: The Leadership Board (LB) ensures that stakeholders are invited to participate in the engagement activities in advance and that communications are appropriate for each stakeholder category. In order to mitigate the risks identified in the previous phase, Engagement Coordination Team (ECT) develops and provides the participants with the briefing materials and coaching needed to ensure the success of the engagement (please also refer to the sub-criterion 3.2).
· Develop an Engagement and Action Plan: The LB, with input from the stakeholders and the support of the Stakeholder Advisory Board (SAB), establishes procedural and behavioural rules for the participants, which may include for example: guaranteeing that the opportunities for providing inputs are evenly distributed among participants, allowing all participants to express their opinion, staying focused on the transformational change that your initiative aims to achieve. Roles and responsibilities for all the participants have to be clearly defined, to regulate their cooperation and allow them to hold each other accountable. Based on the defined mission, create a collective Action Plan (please refer to the sub-criterion 4.1), adopted after consultation with all the participants of your initiative, to guarantee that it corresponds with the expectations of all relevant stakeholders.
In the Review and Improve phase:
· Monitor and review the engagement: The Leadership Board (LB), in cooperation with the Stakeholder Advisory Board (SAB), systematically monitors and evaluates the overall quality of the stakeholder engagement, including the evaluation of (please also refer to the recommendation 5.1.7):
o Commitment and integration;
o Purpose, scope, and stakeholder participation;
o Process (planning, preparing, engaging, acting, reviewing, and improving);
o Outputs and outcomes;
o Reporting.
· Learn and improve: The LB, in cooperation with the SAB and with direct inputs from stakeholders, if needed, continuously assesses the value of the engagement and improves its stakeholder engagement activities for stakeholders’ engagement. These processes need to be formalized to strengthen and optimize future activities.
The stakeholder engagement process is meant to be customized by each initiative which adopts the CRIF, so feel free to adapt and develop it so that it fits your initiative’s specific needs. However, the above-mentioned phases represent the minimum requirements that you have to take into account to implement an effective stakeholder engagement process.
The above-described phases are supposed to be carried out by the Promoters and the LB supported by the Engagement Coordination Team (ECT). This is due to the fact that the first phase (Plan) is expected to be carried out when your initiative is being set-up, while the other activities occur when the Governance Criteria are being implemented, once the LB has been identified.
However, the appointment of the LB itself is carried out through a multi-stakeholder methodology. For this reason, you should follow the recommendations included in this sub-criterion when setting up the LB.
The ECT should support you, as the Promoter, first, and the LB, later, during the entire process that will culminate in the definition of the stakeholder engagement process. This body will also be directly in charge of the implementation of the engagement methodology throughout the development of the initiative.
Sub-criterion 3.2: Engage intended Beneficiaries
Recommendation 3.2.1: Guarantee the availability of customized training for lay participants (patients), who might not be trained to participate in complex research initiatives
Stakeholders such as patients are often involved in a research project as data providers (clinical trials, drug development) or users testing innovative technologies (biotechnological R&I), rather than engaged in the governance of R&I with decision making role. Each initiative adopting the CRIF needs to involve this category of stakeholders to understand their needs and expectations and translate them into practice throughout the entire R&I process. You provide the right tools to all the stakeholders involved, so they are able to equally participate in all the steps of the process.
To successfully engage private intended Beneficiaries, several activities need to be performed. These should be coordinated by the Engagement Coordination Team (ECT), the body that will manage the process of involving several categories of stakeholders including patients, identified in sub-criterion 2.1.1. The main activities are described below:
· Setting in place the engagement process and providing the participants with the necessary briefing materials. These materials should contain a clear explanation of the initiative’s expectations concerning stakeholders’ engagement and facilitate communication between experts and lay participants; the materials should be made available in a timely manner. Make sure that aspects such as linguistic proficiency, disability, and literacy issues of stakeholders are addressed;
· Organizing training sessions in which private Beneficiaries are transparently informed on the process and the role they play within it;
· Guaranteeing the involvement of private intended Beneficiaries that may have experiences in multi-stakeholder initiatives to become the point of reference between the initiative and the stakeholder group.
The ECT focuses not only on the engagement of patients but trains all categories of stakeholders to ensure their fruitful engagement.
Recommendation 3.2.2: Guarantee a fair and equitable process that takes into account the limitations that participants might encounter (e.g. cognitive impairment, behavioural issues, fatigue)
Science with patient input approach requires the active participation of patients in the governance, priority setting, and conducting of research, as well as in summarizing, distributing, sharing, and applying research results. A multi-stakeholder initiative can potentially engage a variety of stakeholders with different levels of expertise, confidence, and experiential knowledge. As explained in the recommendation 3.1.1, it is important to appreciate that some of the stakeholders may face obstacles to becoming engaged by your initiative or contributing to the process to the best of their abilities. Reasons range from lack of knowledge to life-limiting disabilities.
Another essential aspect to be considered is the fact that a research program/project within the health sector can be imagined as a path, namely a sequence of processes and activities in the R&I continuum where patients can be engaged in order to maximize the impact of R&I. R&I Path conceptualizes research as a sequence of processes and activities in the R&I continuum where patients can be engaged in order to maximize the impact of R&I. Consequently, after identifying the possible limitations that might be encountered in the engagement of patients, the Appliers define if these limitations are the same for all patients involved across the R&I Path, or if there are some steps of the R&I Path which are more complicated and for this reason should be considered with more attention.
Following that, the actions to overcome these barriers and limitations need to be envisioned and, if not possible, alternative forms of engagement need to be discussed (i.e. engaging parents for children; relatives of people with cognitive impairments).
The ECT coordinates the participation of patients in the agenda design, in the decision-making process, in the initiative development, and finally in the implementation, monitoring and evaluation phases. Its facilitator role should guarantee that all possible limitations that might affect the effectiveness of patients’ engagement are taken into consideration and that mechanisms to avoid these situations are put in place. Indeed, it is extremely important that the R&I is carefully analysed so that the ECT can be well informed and prepared on the possible limitations that this specific category of stakeholders might encounter in the several R&I steps, and carefully address them to guarantee an efficient and effective stakeholder engagement process. This activity also relates to the Prepare phase of the Stakeholder Engagement Methodology.
Sub-criterion 3.3: Differentiate the level of engagement according to involved stakeholders
Recommendation 3.3.1: Differentiate the level of engagement of involved stakeholders, considering:
· Their skills, capabilities and characteristics;
· The stages and processes of the initiative;
· The relationship with the involved stakeholders and their strategic importance to the initiative;
· The resources available and the organizational constraints
Stakeholders engaged in multi-stakeholder health research initiatives have different skills, expertise, and interests. Once you mapped which stakeholders should take part in the initiative (refer to the sub‑criterion 3.1), cluster them into different categories. Engage stakeholders according to their identified Levels of Engagement. In determining the Levels of Engagement, define the nature of the relationship you will develop with their stakeholders.
Cluster the stakeholders selected by your initiative according to their strategic importance, which could be based on their skills and resources to achieve the initiative’s mission and be accountable.
Stakeholders’ strategic importance for your initiative would then determine the Level of Engagement to be selected to best meet the needs, capacity, and expectations of the relevant stakeholders. Revise the level of engagement periodically, as they may change it over time as relationships deepen and mature. An example of levels of engagement is the following:
· Co-design: stakeholders are engaged since the very beginning of the steps of the R&I Path with a decision-making role (i.e. they are part of the Leadership Board (LB));
· Involve: stakeholders are engaged in research project activities with an active role (i.e. they could be part of the Stakeholder Advisory Board (SAB) with specific roles and/or working groups according to their specific relevance);
· Consult: stakeholders can provide feedbacks to decision-makers on their analysis and/or decisions, and they participate by being asked for advice and opinion (i.e. they could be part of the SAB and/or specific committees);
· Inform: stakeholders are informed about research priorities, activities, outcomes and impact of the initiative.
This prioritization effort will facilitate processes such as the election of representatives of each stakeholder’s category to be part of the LB, advisory bodies, or Working Groups (WGs). It will also be useful during the materiality analysis, when you engage the stakeholders based on the category and strategic importance, among others.
Sub-criterion 3.4: Ensure a balance between engagement of involved stakeholders and agile management of the initiative
Recommendation 3.4.1: Ensure that there is a right balance between an agile management process and the opportunities for engaging a wide range of participants. In particular, set in place processes to mitigate the challenges faced by collaborative groups, such as competition, conflict, cultural and behavioural differences, equity, resource sharing, communication, confidentiality concerns, and geographical dispersion
Identification of appropriate stakeholders to be involved in your initiative is essential to guarantee that there is a balance of different characteristics and backgrounds among participants, which is needed to achieve the transformational change. Moreover, it is fundamental that an initiative prepares appropriate mechanisms to deal with possible challenges that might arise due to the diverse background and characteristics of the stakeholders involved.
To mitigate the challenges that may be encountered by a collaborating group, the Leadership Board (LB), with the support of the Engagement Coordination Team (ECT) and the Compliance Committee (CC):
1. Achieves a balance of interests in the subject matter and in the geographic scope among the participants within the governance bodies;
2. Strives for consensus on decisions that might define the milestones for the initiative;
3. Defines criteria in advance to determine when alternative decision-making procedures should come into effect, in case consensus cannot be achieved. Criteria for determining when to consider voting could include those decision-makers who are not in the agreement. The initiative may want to provide alternative solutions and, if these are not accepted by the majority and a compromise is not reached, then alternative decision-making procedures could be implemented;
4. Defines a decision-making threshold (in relation to the voting process) to ensure that no stakeholder group or type can control the decision-making process.
These two principles might sometimes be in contrast: in this case, the cooperation between the ECT and the LB, with the support of the CC is fundamental to ensure a balance between the engagement of participants and the adaptive management of the initiative.
Criterion 4: Effective and efficient management and coordination of the initiative
· Enables cooperation and competition among participants;
· Sets clear and transparent processes and timeline;
· Maintains flexibility;
· Ensures the presence of secure funding, solid organizational structure, and resource management.
Sub-criterion 4.1: Enable involved stakeholders to coordinate their efforts and perform activities
Recommendation 4.1.1: Put in place processes that allow involved stakeholders to perform mutually reinforcing activities and coordinate collective efforts to maximize results and create opportunities for change
One of the objectives of your initiative is to create accessible and innovative mechanisms to facilitate interaction and bridge the gap between stakeholders to collaborate. Consequently, the Appliers also put in place processes that allow participants to perform mutually reinforcing activities and hold each other accountable through a clear definition of roles and responsibilities.
To allow participants to carry out mutually reinforcing activities, the Leadership Board (LB) should implement the following activities:
· Definition of a collective Action Plan in line with the mission and agenda and specifies the strategies and actions that the different partners commit to implement to achieve such change;
· Implementation of these strategies by all the participants to advance the shared Action Plan;
· Establishment of the Committees and Working Groups (WGs) and other collaborative structures with the role to coordinate activities aligned with the Action Plan;
· Setting up accountability mechanisms to hold partners accountable for implementing activities as planned;
· Organization of Touchpoint Meetings to create opportunities for change, such as:
o Holding periodic events in order to discuss potential challenges, foster innovative thinking, and identify practical solutions;
o Hosting webinars to support stakeholders in the implementation of actions.
The LB is in charge of the implementation of the above actions. It defines the collective Action Plan and oversees that the defined actions are implemented by all the participants. The WGs are composed and balanced according to the stakeholders’ categories and the needs of your initiative. They are in charge of specific tasks (e.g. research or reporting activities, as described in criterion 5). WGs report to the LB. Both cooperation and competition within these bodies should be promoted: participants with different backgrounds, experiences, and interests should be involved in the implementation of a given task/activity to provide multi-disciplinary inputs while pursuing a common goal. This could provide an added value to the initiative itself since multi-stakeholder interactions are considered at all steps of the Research & Innovation Path.
Sub-criterion 4.2: Set clear and transparent processes and timeline
Recommendation 4.2.1: Identify and negotiate with stakeholders a consistent program/project timeline and schedule, in order to assure that the progress is soundly implemented
Recommendation 4.2.2: Commit to transparent, evidence-based decision making, in order to reach the objectives established in the mission and agenda
Recommendation 4.2.3: Guarantee a mechanism of review and evaluation, which allows to learn and improve the collaboration among stakeholders
The Appliers define a timeline to assure that progress is soundly implemented and that the organizational process is transparent. Moreover, they should define clear roles and responsibilities among participants to guarantee that each actor clearly knows their role, exercises their rights, and fulfils their duties. To implement an effective process, the collective Action Plan should also contain:
· Clear and measurable targets to be achieved by the initiative;
· A clear program/project timeline with achievable deadlines to allow participants to hold each other accountable and evaluate the progress achieved by the initiative over time;
· A clear review process which will have to be carried out on a periodical basis to keep track of the achieved targets.
The definition of these rules and deadlines should be discussed and defined by the Leadership Board (LB), because their implementation will be pivotal to guide the initiative in the achievement of its defined mission and agenda.
The implementation of the above activities is strictly related to the previous sub-criterion because WGs are the bodies responsible for carrying out the activities through which the targets can be measured and achieved. To facilitate this process, the LB can appoint a Secretariat or Management Team (please consult Governance bodies section) which will help to enforce deadlines, supervise activities, and improve your initiative’s performance as defined by mission and agenda.
Sub-criterion 4.3: Maintain flexibility
Recommendation 4.3.1: Maintain flexibility, adjusting the goals and implementation actions to the changing reality and needs
It is the task of the Appliers to stay up to date on the current needs of the Beneficiaries. In the implementation phase of the research initiative, they should consider adjusting the goals of the initiative and which stakeholders it engages due to changing needs and reality. When adopting this recommendation, the initiative needs to adapt it to its specific needs and context. Several practices could be evaluated by the Leadership Board (LB) of your initiative to respond to current needs, such as:
1) Prepare a Progress Report (for example on a yearly basis) as it is a useful tool to collect all the achievements but also the concerns raised throughout the process by stakeholders and possible recommendations for the future (sub-criterion 5.1.6);
2) Organize a consultation event on a periodical basis where stakeholders can express their views and confirm their alignment with the defined agenda (sub-criterion 5.2.1);
3) Consider the review by external actors to identify possible gaps and areas for improvements;
4) Periodically review the mission and agenda according to the above-mentioned activities (sub‑criterion 1.2).
These activities could be carried out by specific WGs or other bodies working under the supervision of the governance bodies. We recommend that they adopt flexible risk management. The structure of the initiative and the organization of the activities should be flexible enough to:
· Allow for managing major changes that may arise within and outside the project;
· Guarantee that the initiative is able to pursue the same transformative objective through a different strategy.
It is better to structure the initiative focusing on the objectives, rather than the activities, that may be reviewed following a potential external or internal change and according to the changing scenarios.
Sub-criterion 4.4: Ensure the presence of secure funding, solid organizational structure and resources management
Recommendation 4.4.1: Provide and maintain adequate resources (including financing, staff and technical expertise, and in-kind contribution)
Recommendation 4.4.2: Ensure that the internal team has solid skills to carry out the activities and cooperate with involved stakeholders
Recommendation 4.4.3: Adopt a cost management process and an efficient management to avoid inefficiencies
Recommendation 4.4.4: Maintain accountability over time keeping track of expenses and revenues
For an organization to accomplish its mission and carry out its operations, it is necessary to ensure that it is financially secure. To do so, the organization has to secure funding, create a solid organizational structure with technical expertise, and solid resources management. To implement an effective cost management process, the Leadership Board (LB) may:
· Determine a budget: establish the amount of funding that your initiative has at its disposal;
· Conduct a cost analysis of the project: based on the timeline included in the collective Action Plan, understand the real costs that will be sustained by your initiative throughout the timeline of the project (including research funding, staff, and technical expertise, organization of meetings, other general expenditure);
· Identify possible gaps and critical issues in financial and resource management: identify potential critical issues and develop possible adjustments that would guarantee efficient management of the budget. Identify possible gaps and critical issues based on cost analysis. The analysis should also propose some possible refinements that would guarantee efficient management of the budgeting to avoid inefficiencies.
The LB may choose to appoint a Secretariat/Management Team (sub-criterion 2.2) which will ensure financial security. Depending on the size of your initiative, it could also be supported by other bodies such as the CC and/or others. This process is conducted to ensure that your initiative is financially secure, running public accounting for expenditures and income, and ensuring that it operates in a legally compliant manner in relevant jurisdictions.
Criterion 5: Co-accountability assessment
Appliers establish a shared and effective measurement system, including a set of indicators that promotes the improvement of operations and communication, and set a mechanism to receive feedback. This Criterion is connected to the Materiality Analysis and Master Scorecard (detailed in the respective chapters). To achieve this, the initiative will:
· Develop a shared measurement and monitoring system;
· Establish effective feedback mechanisms;
· Guarantee continuous learning, communication, and disclosure of knowledge.
In this regard, a key step is materiality analysis (sub-criterion 1.2) that enable the initiative to align its activities in coherence with its mission and stakeholders’ perspective.
Sub-criterion 5.1: Define a shared assessment system
Recommendation 5.1.1: Enable the co-selection of relevant aspects, according to the different impact dimensions, in order to identify the topics that matter the most to the initiative and its stakeholders
Recommendation 5.1.2: Select appropriate indicators from the list of relevant aspects according to different impact dimensions and stakeholder perspectives in order to comprehensively assess the impact of health research
Recommendation 5.1.3: Ensure that the list of selected indicators consider the impact on patients
In order to define an assessment system that would be coherent with stakeholders’ perspective and would include the aspects that matter most to them, Appliers consider the aspects chosen via the materiality analysis (recommendation 1.2.1).
In the customised Master Scorecard, the Appliers are able to identify a list of indicators that allow reporting the initiative’s results in relation to different dimensions (efficacy, excellence, economic, social, and patient-reported dimensions). It is important to ensure that the list includes relevant indicators under the dimension Patients Reported Dimensions, indicators that are related to impact on patients directly reported by them without the intervention of the clinicians, such as the Patient Reported Outcome (PRO).
The LB is responsible for the definition of a shared assessment system, however it could nominate a committee to carry out the related activities.
Recommendation 5.1.4: Establish a shared assessment system consisting of a set of indicators consistently tracked over time and a shared data collection process
Recommendation 5.1.5: Ensure that the shared assessment system (Master Scorecard) is coherent to the mission and the agenda of the initiative over time, guaranteeing its alignment to stakeholder perspective
To establish a shared assessment system, the initiative defines a data collection process based on the indicators selected during the Materiality Analysis, which includes all the relevant stakeholders. The indicators need to be consistently tracked over time.
The Leadership Board (LB) ensures that there is constant alignment between the shared assessment system and the mission and agenda of the initiative. Periodically, when the agenda is updated, the shared assessment system needs to be updated as well.
Recommendation 5.1.6: Transparently report and communicate the initiative’s results and progresses to the public
Your initiative communicates its results and progress to the public in a transparent manner, through two complementary solutions:
· A Progress Report published on a regular basis. The Progress Report is a document made available to the public that discloses information regarding the achievement (or non-achievement) of your initiative’s objectives and key performance indicators. In particular, the Progress Report discloses information regarding the indicators identified by the initiative in the sub-criterion 5.1.2, according to the aspects of measurement identified in the sub-criterion 1.2.2. The Progress Report contains general information regarding the management and implementation of the CRIF aspects and other relevant information regarding the achievement of the initiative’s mission and agenda. The Progress Report should be published on a regular basis, every one or two years, according to the specific circumstances of your initiative, and should be published online and made available to relevant stakeholders.
· An open platform, which includes a visualization of the performance of the initiative according to the identified indicators. An open platform is an online tool offering a visualization of the performance of the initiative according to the indicators identified by your initiative (in the sub-criterion 5.1.2, according to the aspects of measurement identified in the sub-criterion 1.2.2). The open platform offers access to key performance indicators regarding the initiative’s implementation. The platform contains general information regarding the management and implementation of the key aspects measured and other relevant information regarding the achievement of the initiative’s mission and agenda.
The Leadership Board (LB) is in charge of gathering information that will constitute the basis for the Progress Report, to create the open platform and to make these tools available to stakeholders and to the public. The LB may appoint a Working Group or a Committee for this purpose or use the help of the Secretariat or Management Team.
Recommendation 5.1.7: Constantly review the initiative according to the results of the assessment
Leveraging the performance assessment requires your initiative to establish a review process that will use its results. The assessment helps in improving performance and practices. To achieve this, the initiative will conduct a periodic review that includes at least the following activities:
· Perform an analysis of the indicators on the initiative’s performance and results, emerging from the shared assessment process;
· Set up an improvement plan identifying counteractions and improvement actions for the initiative;
· If necessary, refine the agenda according to the results of the review.
A third-party actor could be involved in the process to ensure transparency and external oversight. The process should be open to the public to allow external stakeholders to provide suggestions and feedback. It should be implemented on a periodic basis (i.e. every 2 years), according to the needs and the characteristics of your initiative.
The review process should be led by the Leadership Board (LB) and the Stakeholder Advisory Board (SAB), which might appoint a specific committee to carry out the operational activities linked to this process, or depending on the size of the initiative, the Secretariat or Management Team could be as well in charge of developing such activities.
Sub-criterion 5.2: Set effective feedback mechanism
Recommendation 5.2.1: Implement structures and processes allowing to inform, engage, and seek feedback from internal and external stakeholders, including concerns about the initiative and its development
The CRIF attaches great importance to the initiative’s ability and willingness to receive constant feedback from internal and external stakeholders. Both are crucial to improving the efficacy and efficiency of the initiative, not to mention its responsiveness to the ever-evolving needs of stakeholders. For this reason, it is necessary that your initiative establishes a process that allows stakeholders to raise concerns and express their opinions. It is Leadership Board’s (LB) task to:
· Identify the most suitable and appropriate channels through which stakeholders can communicate and raise their concerns (e.g. email, website, letter);
· Set up the activities necessary to gather stakeholders’ feedback;
· Elaborate stakeholders’ feedback;
· Ensure that the feedback is appropriately managed and considered within the review process, under recommendation 5.1.7.
The initiative encourages stakeholders to provide feedback on the implementation of the initiative and keep them informed about the process in place to consider their concerns and integrate their feedback. Channels for feedback may be individual-based (e.g. anonymous hotline, web-format to be filled in) or participative (e.g. working groups, stakeholder consultation processes).
The initiative reports formally on how your initiative analyses, manages, and integrates stakeholders’ feedback. The implementation of this sub-criterion should foster the review process carried out under recommendation 5.1.7.
The LB, supported by the Secretariat or Management Team, is in charge of setting up a process to collect concerns and opinions from stakeholders. The Engagement Coordination Team (ECT) participates in this process and is in charge of maintaining the active participation of internal stakeholders.
Sub-criterion 5.3: Ensure continuous learning, communication and disclosure of knowledge
Recommendation 5.3.1: Establish processes for continuous learning to improve the research evaluation framework and engage the public and the community, building trust among all involved stakeholders through constant communication. Ensure the existence of mechanisms for transparency and prioritize clear, accessible internal and external communication
The Appliers need to build a trustful and continuous relationship with the public and the communities with which your initiative interacts. This can be achieved through a constant, clear, and useful flow of information. To implement this recommendation, the Leadership Board (LB) ensures that:
· Communication on the most salient activities of the project is made public;
· Communication is clear, accessible, and useful. It is made available to stakeholders according to their specific needs.
The initiative can use “unilateral” tools, such as newsletter, website, blogs, reports, but also “interactive” tools, such as training courses, thematic events, peer learning processes, practical guides for users, in-person meetings, events, and others.
The LB is to ensure a constant communication process with other health initiatives that may benefit from (or contribute to) your initiative itself. The LB is in charge of identifying opportunities for information exchange and cooperation and developing the most appropriate means to ensure these relationships in collaboration with the Stakeholder Advisory Board (SAB).
The LB might appoint a specific committee to carry out these activities.
4.4 Baseline Analysis
The Baseline Analysis is a questionnaire that measures the level of compliance of your initiative’s governance and patient engagement with the CRIF. It is recommended that you conduct the Baseline Analysis as soon as you decide to implement the CRIF within your initiative. Performing such self-assessment and learning the results of the Analysis has benefits regardless how advanced the initiative is.
You can conduct the Baseline Analysis via the Toolbox. It contains two sets of questions which evaluate your initiative’s compliance with:
· The five Governance Criteria.
· Existing practices and techniques used in patient engagement (science with patient input, covered in Patient Engagement Guidelines)
The Baseline Analysis tool will automatically provide customized governance recommendations based on the Governance Model Guidelines and Patient Engagement Guidelines, indicating gaps to be addressed. After learning the results, you will know which aspects of your initiative’s governance and patient engagement systems need further development and/or correction. Before starting, take a look at the respective chapter in this Manual with practical tips on how to prepare for maximum efficiency.
5 PATIENT ENGAGEMENT STRATEGY AND GUIDELINES
According to the concept of Responsible Research Innovation (RRI) (European Commission, no date), for research to achieve excellence, validity and relevance, it needs to engage patients and broader society as key stakeholders with decision-making roles. Over the last decade, as health sciences democratize, patient engagement has become more important. Patients started to be engaged not only in a passive role, but also as co-researchers.
Our research concluded that many of the current guidelines for Patient Engagement focus on involving expert patients in the Medicines Lifecycle (i.e. the drug production sequence, from scientific discovery to evaluation).
5.1 Experiential Knowledge
MULTI-ACT proposes a complementary strategy that expands the scope of engagement: a roadmap to capture experiential knowledge of patients (Multi-Act Project, 2020). Experiential knowledge is knowledge gained through experience, as opposed to a priori (before experience) knowledge. It arises when patients’ experiences are converted – consciously or unconsciously – into personal insights that help the patient to cope with the illness. When patients share their experiential knowledge, the collective experiential knowledge exceeds the sum of individual experiences. The notion that patients’ life experience complements researchers’ expertise gains wider recognition nowadays. Patients’ experiential knowledge provides different, yet equally relevant, insights into the R&I. They have a potential of increasing R&I’s impact and producing outcomes that matter most to patients. The experiential knowledge can be harnessed at all stages of the R&I process – from planning to reporting the results.
While collecting patient data has always been important in health research, engaging patients at all relevant stages of your project can enrich the research, enhance its relevance, and ensure that it achieves its goals and brings about outcomes that matter most to people affected by brain diseases.
Patient Engagement Guidelines address the Criterion 2 Participatory Governance and Criterion 3: Clear, effective and inclusive methodology of stakeholder engagement by providing a strategy to empower the stakeholders-patients to be engaged in research & innovation at the same level of the other stakeholders and to empower all the stakeholders to collaborate and co-create with the patients. Compliance with the Governance Criteria is essential for effective patient engagement, and for ensuring return on the engagement for all stakeholders.
5.2 Key assets
The innovativeness of the Patient Engagement strategy relies on three key assets. The first ones are the Governance Criteria and the Engagement Coordination Team. The second – providing training in empowering patients and stakeholders to cooperate and to bring their experiential knowledge into the R&I. It complements currently existing training to make patients “experts”. The final asset is emphasis on the importance of understanding and measuring the impact of R&I. It makes it possible to produce outcomes that matter to patients.
5.3 Science of/with patient input
Science of patient input is about using data provided by people with a disease through passive or active contribution to evaluate impact of R&I. You may think of it as more “traditional” way of doing research, where patients “are studied”. For example, in the context of the CRIF, data about patients’ experiences (Schneeman, Barton and Huneycutt, 2019) outside the clinic are critical to evaluate impact of mission-oriented health research on outcomes that matter most to patients. Science with patient input occurs when patients actively collaborate in the governance, setting priorities, research performance assessment etc. of R&I. It aims to maximize impact of R&I. The concept is relevant to the recommendation 3.2.2.
The figure below illustrates how patient engagement relates to transformational mission and governance bodies. It is important to always ground the engagement process in the mission of your initiative. Governance bodies are responsible for conducting engagement at all stages of the Research & Innovation Path. The Patient Engagement Guidelines are ultimately about raising the return on engagement for your initiative.
5.4 Return on Engagement (RoE)
You can think about return of engagement as akin to return on investment, but your investment is not purely monetary: it is investment in engaging patients in R&I and building relationship with them. The return is considered in broader sense – it is about various impacts and benefits resulting from performing patient engagement in your R&I initiative. RoE is discussed at length in Measuring the performance and effectiveness of patient engagement section of these guidelines, where metrics developed to evaluate whether engagement adds value for different stakeholder groups are described.
5.5 Patient Engagement Roadmap
The Patient Engagement Guidelines provide a Patient Engagement Roadmap for your initiative to capture, understand and draw on patients’ experiential knowledge. You implement the Patient Engagement Guidelines by following this roadmap. We strongly advise you to use the digital Patient Engagement Tool available online in the MULTI-ACT Toolbox (described in the corresponding chapter). The tool will guide you through the following steps:
1) As the Promoter, you are responsible for establishing an Engagement Coordination Team (ECT), the body in charge of management of stakeholder engagement and organize the training modules for the ECT.
2) The ECT defines the phases of R&I Path in which Patient Engagement is instrumental in achieving the mission and agenda of the initiative (see: Research & Innovation Path).
3) The ECT develops Patient Engagement Plans for the steps of the R&I Path identified in the previous action.
4) The ECT identifies indicators to monitor and assess the value and effectiveness of the initiative (Return on Patient Engagement), to verify if it has reached the expected impact on the initiatives.
Roadmap Action 1: Establishment of an Engagement Coordination Team (ECT)
The establishment and training of an Engagement Coordination Team (ECT) is mandatory. This crucial body embeds patient engagement in your initiative’s governance structure.
Establishing the Engagement Coordination Team is a pre-requisite for effective use of the Patient Engagement Guidelines. As a Promoter:
· Ensure that the governance structure, boards and processes of the initiative enable effective patient engagement. Ask yourself: Does the governance structure and process in charge of the Patient Engagement meet the Governance Criteria?
· Define the requirements for appointment of the ECT. It needs to be an open and inclusive process, emphasizing expertise needed from the members to do their jobs. Ask yourself: What is the role and expertise required for the ECT? What training does the ECT needs?
Role of Engagement Coordination Team
The ECT assures that patients feel valued by facilitating their interactions with research teams and creating an inclusive research environment. The team harnesses patients’ experiential knowledge and ensures representativeness of the patients' community. It gathers patients’ feedback. To this end, it needs to be able to translate technical terminology into lay language that patients understand.
The ECT is also responsible for creating commitment among the members and their community. It is therefore its role to facilitate and moderate dialogue between interdisciplinary and different (and sometimes competing) voices and experiences. The ECT sets up your initiative’ dispute resolution system.
Consequently, it also mitigates challenges such as ethical conflicts in protocol design, tokenism, power struggles, difficulties in recruiting different patients, need for additional time, cost.
Skills of Engagement Coordination Team
The following skills as essential to engaging patients effectively:
· Empathy and active listening compassion,
· Communication skills,
· Expertise in engagement strategies & methods (online and offline).
Other preferred abilities and characteristics are:
· scientific knowledge of the disease in question,
· personal experience of the disease as a patient,
· family member or caregiver,
· team work abilities,
· motivational and coaching abilities,
· socio-psychological knowledge/background,
· ethical management knowledge/background,
· understanding of group dynamics,
· project management knowledge/background.
Composition of the Engagement Coordination Team
Below is an example composition of the ECT. It can be modified to fit your initiative’s situation. The important thing is to ensure that all key stakeholders and represented and that
· Co-Chair, patient (1 person),
· Co-Chair, MULTI-ACT trained representative (1 person): this individual has to complete the training (see Training of Engagement Coordination Team members),
· Initiative’s board representative (1 person),
· Initiative’s staff representative (1 person),
· Patients (with consideration to the balance of gender, geography, disease progression, age, socioeconomic background) (3-6 persons),
· Expert(s) on the mission and priorities of the initiative, e.g. Working Group representative, industry forum representative.
Training of Engagement Coordination Team members
The ECT is expected to be a unique board of experts with innovative functions, knowledge and expertise. If they are a new team, they will require innovative training. In order to allow the ECT to integrate patient experiential knowledge in research of your initiative, you should design and provide to the ECT a training module. Make sure that it includes:
· Adequate information about the project's mission and strategy;
· Explanation of what is expected from patients and other stakeholders;
· Explanation of what are the expected outcomes of the multi-stakeholder initiative;
· Explanation of how these outcomes relate to the patients' needs in the given disease area;
· Basic knowledge about innovative communication, learning and co-working techniques, and evidencing the value of patient and stakeholder engagement.
Information on methods for patients’ engagement should be integrated with examples of application in real cases for each method. The training should focus on the ability to elicit and capture patients’ stories and translate them into experiential knowledge. Use plain language and keep the content simple. Respect for human rights and dignity of the patient should always be considered.
Roadmap Action 2: Selection of research priorities and steps where patient engagement is instrumental to meet the Mission
Although patient involvement is crucial at every stage of the research, it is advisable to verify in which steps of the Research &Innovation Path (R&I Path) it is best to engage patients to maximize the impact of your research.
As described in the Plan phase in the sub-criterion 3.1 and in the recommendation 3.2.2, it is equally important for your initiative to identify obstacles that the patients may face in becoming fully engaged and contributing. The Criterion 3: Clear, effective and inclusive methodology of stakeholder engagement of the Governance Criteria looks at involving stakeholders (including patients) from a broader, governance a perspective. We advise that you look into the engagement-related governance recommendations alongside the Patient Engagement Guidelines.
Research & Innovation Path
Research & Innovation Path (R&I Path) is a sequence of processes and activities in the R&I, in which patients can be engaged in order to maximize the impact of the research initiatives. The steps of the R&I Path represent stages in research and management of funding and performing research within initiatives or in projects conducted by Research Funding and Performing Organisations (RFPOs). The steps of the R&I Path differ slightly for the Governance Program Level, which concerns often complex research programs comprising of multiple projects, and for Project Development Level, which concerns single research projects. Although patient involvement is considered crucial at every stage of the research, it is advisable to verify in which steps of the R&I Path it is best to engage patients to maximize the impact of your research.
Detailed descriptions of the steps for each level are presented in the tables below.
|
Governance Program |
|
|
Breaking down the boundaries |
Conditions that should be set in RFPOs in order to facilitate patient engagement as standard practice, e.g. patients help to review patient engagement policies and guidelines. |
|
Setting research priorities |
Actions to raise interest in a specific research domain, its importance, priority or rank. E.g., patients advance their interests in a specific research area. |
|
Steering institutions |
Actions performed to establish governance bodies. E.g. patients are invited to be member of committees and boards. |
|
Design and planning |
Design and planning of all the activities that lead to implementation of a concept or idea, and which help to achieve initiative’s designated objective. Patients are engaged in the development and monitoring of research programs |
|
Executing research |
Activities to perform the research program or a specific research project for the purpose of achieving the initiative’s designated objectives. Project Development (see below) level takes places at this stage. |
|
Evaluating research |
Activities to determine the value created by a research program or project, establishing their outputs and outcomes, the degree to which their pre-established goals were achieved, and their impact. Patients are engaged to working with other stakeholders on research reports. |
|
Translation to community |
Activities to foster and facilitate the uptake of results of research programs or projects within wider society. Patients are engaged in the development of guidelines and advocacy activities. |
|
Project Development |
|
|
Design & plan |
Design and planning of all the activities that lead to implementation of a concept or idea, and which help to achieve initiative’s designated objective. Patients are engaged in the development and monitoring of research programs. |
|
Conduct & operate |
Conducting & monitoring project (e.g. ICT device development). |
|
Evaluation |
Activities to determine the value created by a research program or project, establishing their outputs and outcomes, the degree to which their pre-established goals were achieved, and their impact. Patients are engaged to working with other stakeholders on research reports. |
|
Translation to community |
Activities to foster and facilitate the uptake of results of research programs or projects within wider society. Patients are engaged in the development of guidelines and advocacy activities. |
Levels of Engagement
Stakeholders can contribute to the health research and innovation simply by getting informed or by participating in your initiative with various levels of decision-making power. Levels of Engagement presented below are a useful way of describing the varying depth of patient engagement with research and innovation (R&I) process. The same stakeholder may be engaged at different levels depending on the phase of the initiative, their role, or other factors. Levels of Engagement are relevant to sub‑criterion 3.3 of the Governance Criteria.
|
Co-design
|
Stakeholders are engaged since the very beginning of the R&I initiative with a decision-making role. Examples: Patients are asked to co-define the share agenda and co-design research governance. Stakeholders are members of the Leadership Board. |
|
Involve |
Patients are engaged in the research initiative and given an active role: they provide their perspective and/or data on a specific topic. However, the initiative is designed and initiated by professionals and researchers. Examples: Gathering patients’ views on the topics that are important for them. Co-creation of the patient-reported outcome measurements for clinical trials development. Stakeholders act as members of the Leadership Board or Working Groups. |
|
Consult |
Stakeholders are asked to provide feedback for decision-makers about their analysis or decisions. Stakeholders participate by being asked for advice and opinion, by expressing their views and having discussions. It does not usually include any share in decision-making. Examples: Consulting activities, survey, interviews, establishing and maintaining relationship with stakeholders. Stakeholders act as members of the Stakeholder Advisory Board. |
|
Inform |
Stakeholders are informed about research priorities, activities, outcomes and impact. Patients receive information from researchers in a passive way. |
Roadmap Action 3: Design and implement a Patient Engagement Plan for each identified research priority and step
Once defined and agreed, the R&I Path steps where patient engagement is instrumental to achieving the mission, the Engagement Coordination Team designs and implements a Patient Engagement Plan for each identified research priority and define Patient Engagement Actions for each step. The Plan should include:
· Selected actions of patient engagement that needs to be implemented in order to achieve the vision of the project;
· Roles and responsibilities of the team that should manage and carry out the implementation of such Patient Engagement actions;
· Methods to value and acknowledge the experiential knowledge of patients, including the establishment of appropriate recognition of patient contribution, and avoid tokenism;
· Measurable targets (measuring the performance and Return on Engagement);
· Timeline of activities and sustainable budget;
· Review process (e.g. report on the performance and Return on Patient Engagement).
Below you will find the Menu of Patient Engagement Activities: these are suggestions of patient engagement activities suitable for each of the R&I Path steps. You are encouraged to create your own activities.
|
7-steps R&I Path |
Menu of Patient Engagement Activities |
|
BREAKING DOWN BOUNDARIES |
Patients help to define what are the boundary condition for patient engagement in your multi-stakeholder initiative. Patients help to provide an overview on the facilities and infrastructure they need to be engaged in the R&I. Patients help to review patient engagement policies and guidelines. |
|
RESEARCH PRIORITIES |
Patients are engaged to co-design research agenda. Patients are engaged in advancing their interests in a specific research area. Patients are engaged to prioritize research objectives. |
|
STEERING INSTITUTIONS |
Patients are invited to be members of committees and boards; they provide guidance on key issues such as company’s policy and objectives, budgetary control, marketing strategy, resource allocation, and decisions involving large expenditures. Patients are invited to advise the steering and advisory committees. Patients are engaged in defining ethical issues, anticipating risks and barriers for patient engagement in governance bodies. |
|
DESIGN & PLAN |
Patients are engaged to suggest endpoints and outcomes of research. Patients are engaged to propose specific objectives of research. Patients are engaged to define the relevance and acceptability of proposed research to patient community. |
|
RESEARCH EXECUTION |
Patients are engaged in the development and monitoring of research projects (e.g. collaborating for ICT device development, for the enrolment to increase participation and decrease drop-out rate, to increase compliance with protocols and facilitate data collection, for writing and review of publications. Patients are engaged in development and monitoring of research programs (e.g. release of calls for proposals, selection of projects to be funded, monitoring of funded projects). |
|
EVALUATION |
Patients are engaged in discussions in multi-stakeholder teams about new methods to measure the impact of research. Patients are engaged in assessment of new approach and products arising from research. Patients are engaged to working with other stakeholders on research reports. |
|
TRANSLATION TO COMMUNITY |
Patients are engaged in shaping the ‘translation strategy’ of research results into easy-to-use and easy-to-understand (lay) material and in communication activities to disseminate the research results. Patients are engaged in the development of guidelines and advocacy activities Patients are engaged in advocacy to leverage uptake of the research results. |
Patient Engagement Plan
The Patient Engagement Plan is a framework that allows your initiative to plan patient engagement in a systematic manner consistent with progress towards fulfilling the mission. The MULTI-ACT Guidelines offers a practical template to support design of the Patient Engagement Plan: you can find it in the Appendix 3: Patient Engagement Plan Template and/or a digitally function to develop Patient Engagement Plan in the Toolbox. Its purpose is to provide you with a tool to facilitate the design of operative patient engagement plans that are compliant with the MULTI-ACT guidelines. The Plan is structured to integrate patients’ experiential knowledge into your R&I initiative, bringing expertise and knowledge complementary to the ones of other stakeholders. Patients, as members of the ECT and key stakeholders, develop the Patient Engagement Plan with the other stakeholders, ensuring representativeness of their community.
The assessment of the implementation of the Patient Engagement Plan is aligned with the Plan phase in the sub-criterion 3.1.
The indication included in the template found in the Appendix 3: Patient Engagement Plan Template is not necessarily the norm or a common practice, but rather a first attempt to provide practical guidance to RFPOs on how to plan, launch and monitor their Patient Engagement actions. Each mission is unique and requires ad-hoc interventions.
We recommend using the Patient Engagement Plan tool in the digital Toolbox that facilitates you with drop-down options and suggestions for compiling each field.
Methods to engage patients
The following paragraphs contain recommendations on methods of patient engagement. They align with the Prepare phase of the sub-criterion 3.1.
Creating right conditions
It is important that the ECT establishes a supportive research environment which leverages patient engagement (communication channels, resources, infrastructures, organizational/institutional). The ECT needs to assure that patients understand and agree on the research agendas, and to assure that they feel comfortable and recognize that their perspective is unique. The ECT also has to strengthen the team spirit by creating a supportive environment that promotes partnership and open dialogue.
You can find more tips and principles to follow when engaging this special stakeholder category in the recommendations 3.1.1 and 3.2.2 and in the Patient Engagement Guidelines.
Using the right methods
In practical terms, the best way to engage patients is to use mixed methods: offline (face-to-face) methods (engagement without using computers, smartphones, tablets, or other internet-connected device/digital systems) and online methods (engagement through computers, smart phones, tablets, or other internet-connected device/digital systems).
Online methods make it possible to gather patients’ perspectives on a global scale while offline methods are useful to facilitate patients in providing their experiential knowledge as they may feel more comfortable to express their feelings face-to-face and they may be supported by a professional skilled managerial team (i.e. the ECT). Moreover, offline methods allow stakeholders to discuss more in-depth and to establish and to maintain a good partnership with patients. In particular, the ECT works mainly offline and they may use online methods to reach their community and a large consensus.
IT tools are useful for joining and recording conversations, proactively resolving complaints, promoting transparency, and enhancing patient experiences. It also requires organizations to comply with meaningful use criteria, such as engaging patients and families in their care, improving quality and care coordination, and reducing disparities (Thielst, 2011). Use of social media may affect patient engagement and satisfaction in healthcare and research. Integration of social media into clinical practice and research can empower surgeons to synthesize effectively a patient support community that increases patient engagement and satisfaction (Dhar et al., 2018). The same may apply as well to the R&I domain and environment.
Social media may play a role in identifying patient insights and engaging them in R&I for the purpose of capturing their experiential knowledge. Evidence related to the efficacy and effectiveness of social media in this function is currently limited. Various challenges related to privacy and security concerns, usability, the manipulation of identity, and misinformation have also been identified (Househ, Borycki and Kushniruk, 2014). You have to exercise caution in their use and investigate, whether the way to envision employing them for patient engagement has a scientific basis. Use of social media and social networks for science and research as a method to capture patients’ voice is worth investigating for a start (Musso et al., 2018; Fontaine et al., 2019), even though it has not been scientifically validated.
Review and ranking of the most appropriate methods for the engagement of patients and other stakeholders
Below you will find descriptions of the methods selected by MULTI-ACT as appropriate for the engagement of the public in decision-making processes in the R&I, and in particular of patients. These methods are the following: Focus Group, Democs Card Games, World Café, Consensus Conference, Community Advisory Board, Delphi Method, Citizens Hearing, Serious Gaming. Many of the methods have a versatility to be used both online and offline.
The list is by no means exhaustive; your choice of the method depends on:
· the goal of the specific engagement event – what you want to get out of it,
· the stakeholders being engaged: how much time and effort they can contribute, what obstacles they may face,
· what resources you have assigned for the occasion (monetary, human, time etc.),
· how familiar you are with the technologies to be used.
Focus Group and Democs Card Games are useful for capturing experiential knowledge and give voice to Patients. In line with the CRIF, these two methods are considered "good methods" that the ECT could apply to engage patients and stakeholders in R&I.
|
Focus Group |
Focus Group is a qualitative method which is used to determine the preferences of people or to evaluate strategies and concepts. The method has originally been designed for market research. Focus group is undoubtedly the most widespread technique of engagement. It is rooted in qualitative studies, where it is a standard way of gathering patients’ input and learning about their views and experiences. Its scope of application has widened in recent years, with the method being used for decision-making and guidelines formulation (Doria et al., 2018), not without some criticism regarding insufficient separation of these two functions. Participants are selected according to certain common characteristics that relate to the research topic and are grouped into 8-10 people. It can be conducted face to face or in virtual digital space. The method is often used to generate or evaluate hypotheses and ideas in conjunction with a quantitative method, or as a primary data-collection method. Example: Selected patients and stakeholders are invited to a meeting to discuss about a topic. |
|
Democs
|
It is both a card game and a policy-exploration tool that enables small groups of people to engage with complex public policy issues. It aims to help people find out about a topic, express their views, seek common ground with other participants, and state their preferred policy position. There are already a number of Democs kits on different issues which can be bought or downloaded for free from New Economics Foundation (NEF) and Play Decide. Example: Patients are provided with discussion cards that help them to express their views on a topic, to seek common ground with the other participants, and to express their preferences. |
In the Appendix 5: Patient Engagement Methods, there is a brief presentation of other suggested methods, based on the descriptions from the Engage2020 – Action Catalogue.
Roadmap Action 4: Selection of the indicators to be used to measure the success and effectiveness of this engagement
Measuring the performance and effectiveness of patient engagement
To maximize the impact of patient engagement, the ECT identifies indicators suitable for performance measurement and assessment of the effectiveness of patient engagement in your initiative’s R&I processes. The assessment should combine quantitative and qualitative evaluation. The assessment of the implementation of the Patient Engagement Plan is a part of the Review and improve phase in the sub-criterion 3.1: Define and approve a methodology to engage stakeholders.
For patients, the most important benefit from the engagement in the R&I is its influence on the outcomes that matter most to them, such as their care, treatment, quality of life, and how they feel about their symptoms and/or functions.
Performance of patient engagement is about the success of your initiative in terms of participation. The associated indicators are to be selected ex-ante (before), included in the Patient Engagement Plan and verified ex-post (after) the development of the plan. The additional indicators should be considered examples only.
|
Core indicators |
· Number of different phases of the research process (Patient engagement in the Research & Innovation Path) patients were engaged in. · Number of patients engaged across different socio-economic statuses, education backgrounds, genders, etc., to assess the capacity to engage diverse groups, including the most vulnerable ones. · Number of engagement actions (online and offline) that took place, in which patients had an opportunity to express their views. |
|
Additional quantitative indicators |
· Number of KPIs selected to assess the impact of patient engagement. · Number of conducted training. · Extent into which the patient involvement at the end is implemented in the research path. · Number of interviews about patients’ experience in the engagement process. · Number of co-created tools the engagement measurement. · Number of reality test made by the patients. · Number of patient 'intervention' directly, or indirectly. |
|
Additional qualitative indicators |
· Analysis of the patients’ expectation with respect to the patient engagement are met. · Analysis of whether the patients have felt engaged, listened and valued. · Analysis of how meaningful the engagement was to the patients as well as to the research team. · Analysis of how patients have been engaged (e.g. collecting comments, surveys, feedback, etc.) |
Effectiveness of patient engagement is about success of your initiative in term of real impact of the participation on the research process: whether the actions performed have effectively produced impact and change in the R&I process. The indicators of effectiveness are included in the Digital Toolbox as a sub-set of the Patient Reported Dimension. You can also find the in the Appendix 4: Master Scorecard.
|
Core indicators |
Quantitative · Number of changes in the research process (e.g. policies, composition of boards, objectives and priorities, strategic plan, evaluation of results, dissemination actions, etc.) according to the review made by patients. · Number of research initiatives, programs and/or projects that include and show an effect on Patient Reported Outcomes (i.e. questionnaire reporting how they feel about symptoms and functions). · Number of research initiatives, programs and/or projects involving patients in research activities, according to the needs of the mission. Qualitative · Analysis of whether patients’ expectation with respect to the research and mission of the initiative are met. · Evidence of the satisfaction and endorsements given by patients to research outcomes and results. · Evidence on patients’ satisfaction with their engagement in the research in terms of expectation and influence on research outcomes, including identification of benefits and critical issues (pros and cons), and need for implementation. · Analysis of the achievement in terms of new knowledge produced, from the perspective of all the stakeholders. |
|
Additional quantitative indicators |
· Number of patients engaged in research activities, according to mission’s requirements. · The degree of representativeness: the number of the underrepresented population and of the disadvantaged patients involved in the research. · Number of dissemination actions carried out by patients (e.g. events where patients presented and endorsed research results). · Number of scientific articles in which patients are co-authors and/or reviewers. · Number of endorsements given by patients to research activities and results. · Number of endorsements given by patient organisations. |
|
Additional qualitative indicators |
· Analysis of how patients’ lives may be or have been improved by the research. · Analysis of the long-term improvement in health indicators. · Analysis of whether the value of patient contribution is the same as other stakeholders. · Evaluation of the research initiative, program and/or project plan, of all single research phases and of the results, by patients and if and how their suggestions has been integrated into the research activities. |
6 COLLECTIVE IMPACT ASSESSMENT
Co‑accountability lies at the centre of the CRIF: stakeholders agree on a common mission and a shared agenda. Subsequently, they co-create a common impact assessment system for measuring their progress towards the agenda. By “impact” we understand changes in the world (e.g. for the society, for patients) that happened because of an initiative’s activities. MULTI-ACT defines impact as long-term (over 5 years) socio‑economic changes the intervention brings about, as opposed to “outcomes” which are more short-term. Impact assessment is a mean to measure the effects/changes/results that the initiative brings about. It includes conceptualization of the causal relationships between what inputs and impact, i.e. the research and other activities of an initiative and changes in the society (health improvement, higher well-being etc.). Additionally, some measure of both the activities and changes is needed. In the co‑accountability framework, the indicators are not static but subject to change across time in order to properly respond to the changing environment and to the changing stakeholders needs.
This chapter discusses the MULTI-ACT tools your initiative can use to establish the common impact assessment system and the system itself:
· Through the Materiality Analysis, you engage all categories of stakeholders in selecting indicators for a customised assessment system based on their materiality
· The Master Scorecard provides the indicators to choose from, at the same time covering all relevant aspects of impact
· Additionally, Patient-reported Outcomes allow you to measure the impact of your research on the patients.
6.1 Materiality Analysis
Not all changes brought about by an intervention or research initiative can be taken into account, and not all are equally relevant and significant. Choices have to be made about which data is tracked and reported, and when monitoring is optional: that is how impact indicators are chosen and created. Generally speaking, a piece of data is material if its omission or misrepresentation may affect stakeholders’ decisions or their ability to draw reasonable conclusions about the impact. It may be useful to think about materiality as a threshold above which missing or misrepresented information is considered to have an impact on the decision making.
Naturally, organizations and individuals differ in their decisions on what is material, depending on their sector, mission, vision, values, strategy, dominant stakeholders and background.
In order to achieve co-accountability, stakeholders in your initiative participate in selecting of the indicators. The process is called materiality analysis. CRIF makes it easy to conduct the Materiality Analysis via the Toolbox.
Within the CRIF, the materiality analysis is a process through which your initiative’s stakeholders will determine which indicators the initiative will use for assessing its impact. Materiality analysis is a step towards co-accountability, as representatives of all stakeholders categories within your initiative will be engaged in selecting the indicators. The representatives will be collectively held responsible for monitoring and reporting these parameters. They will give their judgements on which aspects and indicators of the Master Scorecard are both:
· Relevant to your initiative’s mission and agenda and
· Significant enough to be considered material, i.e. their inclusion or omission may influence decision-making
Therefore, it is a prerequisite to creating your customised Master Scorecard and to conducting impact assessment. The materiality analysis is a “bridge” leading from the Governance to the Impact Assessment part of the CRIF.
In the Materiality Analysis, you make use of information that you have already provided in the Baseline Analysis: mission and agenda and stakeholder engagement. If your initiative has not yet formulated mission and priorities, you need to conduct the Baseline Analysis before starting the Materiality Analysis. Governance guidelines in the Criterion 1 will help you to do this. In the Criterion 2, you will find guidelines for effective stakeholder engagement.
It is your task, as the Promoter, to start the process of materiality analysis in the Toolbox. These are some general recommendations to be followed to get a robust and reliable analysis:
1) Cluster the responses by different stakeholder categories. Results can be then aggregated following the suggested methodology presented in the box below
2) Ensure anonymity of the responses.
3) Define a minimum number of individual views required to be considered representative of a stakeholder category (e.g. minimum 4), in order to ensure a balanced and veridical representation.
4) Try to reach a heterogeneous cluster of responses within the same category: introducing additional specificities inside each stakeholder category helps catching potential differences within the same cluster (i.e. perspective of patients who are under treatment and not under treatment).
5) Provide complete guidelines and/or tools to respondents that may not be fully aware of the initiative and the CRIF.
6) Clarify the threshold under which the responses are considered non-representative and thus inadmissible.
6.2 Master Scorecard
The Master Scorecard is an adaptive tool for assessing health research & innovation initiatives and projects. It applies a multi-stakeholder perspective to provide a list of indicators for the assessment of research impact, considering the five CRIF dimensions (excellence, efficacy, social, economic and patient-reported). The scorecard provides the indicators to evaluate the impact of health research and innovation, paying special attention to the benefits to patients, healthcare and society in line with the multi-stakeholder initiative’s mission. More specifically, its purpose is to offer an innovative and multi-perspective approach for the organization to gather information on the achievement of its objectives concerning its impact.
It provides a catalogue of 125 indicators grouped into five CRIF dimensions. For each indicator, the scorecard offers its description, example, qualitative or quantitative measurement, and methods and data sources, among other details.
CRIF Dimensions
Conventional metrics of academic excellence in research were integrated with indicators of economic impact, financial balance, social influence and – last but not least – measures of capacity to accomplish one’s pre-defined mission. Patient-reported dimension overarches the other four dimensions, introducing the perspective of patient as the key stakeholder to impact assessment as a whole.
CRIF impact assessment comprises five dimensions that reflect different accountability perspectives: efficacy, excellence, economic, social and patient-reported that are described below.
Efficacy dimension
Efficacy dimension looks at your initiative’s capacity to accomplish the mission it defined for itself. This dimension is the main driver for co-accountability within CRIF, because it assesses to what extent the initiative brings value for its stakeholders as pre-defined in the mission. In the context of brain research, the mission focuses on the improvement of the life conditions of patients affected by brain diseases, while balancing the conflicting perspectives of the different stakeholders involved. This dimension has the strongest links to Governance Criteria.
Excellence dimension
This dimension focuses on the quality of scientific research that is conducted as part of you initiative. While it contains traditional bibliometric indicators used to measure academic performance, it goes beyond them, allowing for appraise contribution to knowledge and impact on society. The indicators reflect MULTI-ACT conviction that research should positively influence people’s lives to be deemed “excellent”.
Economic dimension
The economic dimension contains a set of economic and financial indicators. Monitoring your initiative’s internal financial balance is crucial to make it sustainable in the long run. Estimating its influence on the economy, e.g. through keeping patients in the workforce, is important for demonstrating its social impact in holistic manner.
Social dimension
In implementing social dimensions indicators, you are encouraged to look at the long-term direct and indirect effects of your initiative on the society as a whole, beyond primary stakeholders and Beneficiaries. It also includes communication with the society (e.g. external reporting) and community engagement.
Patient-reported dimension (PRD)
Patient-reported dimension is the transversal one, in which the other four are rooted. It places patient at the centre of health research as the key stakeholder, whose needs and perspectives must be understood and incorporated into the research process. It is a tool enabling the Science of Patient Input since it includes indicators that are reported by patients. PRD comprises two groups of indicators:
· Patient Reported Outcomes (PROs) and Patient Reported Outcomes Measures (PROMs),
· Qualitative indicators to assess the Return on Engagement (RoE).
CRIF Aspects
Indicators within each dimension are grouped in order to make them more manageable, both at the stage of selection (Materiality Analysis) and later on, when the initiative implements them and uses them in monitoring its impact. They reflect assessment perspective.
Aspects are broader categories, and groups – more specific, narrower categories of indicators. In each aspect, there is at least one core indicator.
CRIF Indicators
The Master Scorecard provides a catalogue of 125 indicators, grouped into 5 dimensions.
|
Dimensions |
Aspects |
Indicators |
|
Patient-reported |
9 |
11 |
|
Economic |
9 |
20 |
|
Efficacy |
9 |
22 |
|
Social |
6 |
15 |
|
Excellence |
20 |
57 |
|
Total |
53 |
125 |
Core and additional indicators
Not all indicators are equally important for each initiative. Core indicators are obligatory to use. There is at least one core indicator per CRIF aspect.
|
Dimension |
Core indicators |
Additional indicators |
|
Patient-reported |
9 |
2 |
|
Economic |
9 |
11 |
|
Efficacy |
9 |
13 |
|
Social |
7 |
8 |
|
Excellence |
20 |
37 |
|
Total |
54 |
71 |
The CRIF recommends your initiative to select additional indicators that will increase its co‑accountability in an aspect that you consider material, or in situations when your initiative is not able to apply a related core indicator, e.g. due to lack of required data. Indicators were selected and divided into these categories based on extensive literature review.
Qualitative and quantitative indicators
Indicators in the Master Scorecard, irrespective of their core or additional status, are of either qualitative or quantitative nature. This distinction will help you to figure out what kind of input is expected from you when you fill in each particular indicator in the Toolbox. For qualitative indicators, you are expected to provide a narrative description and/or mark your answer to a qualitative question. When it comes to quantitative indicator, some kind of numerical input is required.
Using the Master Scorecard
The indicator can be applied for impact assessment at the beginning or during the development of a research initiative to select the indicators through the approach described in the Materiality Analysis. Depending on the stage of the project life cycle, the Master Scorecard can serve different purposes:
|
Initiation |
Planning: The CRIF dimension (and the potential indicators) developed with the Master Scorecard allows the research initiative to (ex-ante) strategically design and evaluate the expected impact of a research project, according to its vision and agenda. |
|
Execution |
Monitoring: the Master Scorecard can serve to implement the mission selected by the initiative, in line with its vision and agenda. It can be used as a monitoring tool to assess the research and innovation activities delineated by CRIF dimensions. It could be used iteratively during the execution of the initiatives with the appropriate frequency. |
|
Closure |
Assessment: the Master Scorecard can be applied at the end of the initiative in order to assess how the desired results were reached. If the Master Scorecard is applied from the beginning of the project, the impacts can be compared with the initial evaluation output. This can help to strategically orient also future initiatives. |
The Master Scorecard enables strategic management of multi-stakeholder research initiatives. It assists initiatives in the evaluation of the multiple dimensions and impacts of their health research and innovation activities. It can be used as a strategic management tool as it helps to monitor the progress of research and innovation projects and to demonstrate whether and how the initiatives are producing actual outcomes and impacts. However, the user must be aware that the computation of the indicators selected by an initiative from the scorecard will not provide an overall “score” or “ranking”.
The Master Scorecard can be used in managing initiatives’ operations, identifying outcomes of the research, or for controlling and improving the initiative's performance.
The Master Scorecard is useful for initiatives aiming to increase the impact of research on people and society. The multi-stakeholder nature of MULTI-ACT allows the engagement of a broad range of users, which can be engaged in customizing and applying the Master Scorecard.
The Master Scorecard can be adapted to many individual needs:
· It allows flexibility and can be tailored to diverse multi-stakeholder projects, so it should not be used as a fixed set of indicators. It offers a starting point to be applied and tested in different contexts and settings, especially to multiple sclerosis or other brain diseases.
· It is dynamic as you can select indicators for different purposes and specific needs of many stakeholders through Materiality Analysis.
It is constructed in a way that it can be used, customized and applied by a broad range of users. It is an indicator catalogue that covers a wide range of relevant aspects that can be used in assessing the multiple impacts of health research. Therefore, initiative can select indicators among different topics and possibilities according to their own needs.
Your initiative can adopt Master Scorecard to build co-accountability by linking the research outputs with the mission and priorities of the initiative. It can be done regardless of the stage of R&I initiative, although early adoption renders best results.
6.3 Patient Reported Outcomes (PROs) and Patient Reported Outcomes Measures (PROMs)
PRO is any report about patient’s health status coming directly from the patient. This report cannot be used or interpreted by anyone else (FDA, 2009). PROs are strictly about patient’s perception of disease and treatment (European Medicines Agency, 2014), so they hold a special place within CRIF, where the patient is the key stakeholder, contributing their experiential knowledge to the research.
PROMs are standardized, validated questionnaires (which are also called instruments) completed by patients to measure their perception of their functional well-being and health status (Department of Health, 2009). PROMs are questionnaires measuring the patients’ views of their health status. PROMs are used to assess a patient’s health status at a particular point in time. PROMs tools can be completed either during an illness or while treating a health condition. In some cases, using pre- and post-event PROMs can help measure the impact of an intervention. PROMs are tools used to measure patient‑reported outcomes (PROs). PROMs are offered in the Toolbox.
7 DIGITAL TOOLBOX
The digital MULTI‑ACT Toolbox is an online platform that will assist you in implementing CRIF. While it is possible to implement CRIF without using it, it is much easier to use it, and this Manual is based on the assumption that you do. Using the Toolbox is free. It is available at: https://toolbox.multiact.eu
The Toolbox is designed to be intuitive in use and it contains a wealth of tips and explanations at each step. It will often refer you to relevant parts of the CRIF Manual. Therefore, in this chapter you will find only general introductions to the Toolbox functionalities. The main tools in the Toolbox are:
· Baseline Analysis
· Patient Engagement Plan
· Materiality Analysis
· Master Scorecard
· Patient-reported Outcomes
7.1 Creating an account
If you have not yet set up an account, go to “log in” in the right upper corner of the page. You will be taken to a page where you can log in, create an account and reset your password.
7.2 Your user account
Once you create your account and log in, you get access to all the functionalities. 
Below you will find explanations of the functionalities marked with numbers
 |
Wherever you are in the Toolbox, you can always go back to your account. |
 |
At any time, you can go to the Master Scorecard browser. Therein, you can read detailed information about each indicator from the Master Scorecard: related literature, examples, expected data collection frequency, expected reporting frequency, limitations, unit, method of measurement, and many more. |
|
|
This Manual is available both as a .pdf document and as a sub-page on the Toolbox website. The Toolbox will often refer you to a relevant section of the CRIF Manual when explanations are needed. Feel free to use the format you prefer. |
|
|
This is the place where you can change your password, your e-mail and other registration data. |
|
|
In here, you can see invitations to become a member. To invite others to be members of your initiative, you have to create an initiative and enter the “invitations” functionality from there. |
|
|
Once you create initiatives, they will be listed here. You can enter each of them by clicking on its name. |
|
|
To start the process of creating a new initiative, click here. |
7.3 Creating an initiative
Once you name your initiative, you will be asked to add basic information about it. You will be asked to provide basic information about your undertaking, and you will be given a chance to upload documents that will be useful later on. After you do it, you will get access to the initiative overview page.
As with the user account, get to know the functionalities of the account.
|
|
You can change the name of your initiative here. |
||
|
|
This functionality allows you to send invitations to other members of the governance bodies who will manage the implementation of CRIF alongside you. Members of the Engagement Coordination Team (ECT) are especially important, since it is they who are responsible for the preparation of the Patient Engagement Plan. Be sure to invite them early on and encourage them to read the CRIF Manual. |
||
|
|
Invite one member by introducing their e‑mail. |
||
|
|
Invite multiple members by introducing their e‑mails in bulk. |
||
|
|
In the “members” section, you can see the invitees who have accepted your request. |
||
|
|
Appoint members to specific roles in the initiative. |
||
|
|
Throughout the CRIF Manual, you will see the roll-down texts which can assist you at each step. It is worth it to read them. |
||
|
|
The next step after filling the initiative information is to conduct the Baseline Analysis. The Patient Engagement Plan tool will appear only after you will have finished the Baseline Analysis. Similarly – the Materiality Analysis appears only after accomplished Patient Engagement Plan. |
||
7.4 Baseline Analysis
During the process of filling in the Baseline Analysis questionnaire, you will be asked to upload various documents: financial reports, yearly reports, sustainability reports, internal policies on patient engagement, mission and vision, ethical compliance, monitoring and evaluation, social and environmental impact assessment, governance bodies and management procedures, academic achievement etc. While it may require effort to collect the documents, it will pay off as the Baseline Analysis results will help you identify gaps in governance of your initiative and align different procedures with the mission. The Toolbox will ask you to categorize the documents you will upload and sources you will refer to. You may find Appendix 1: Documents classification helpful.
When you fill in the questionnaire, you will receive your score and accompanying recommendations.
You can learn your compliance status for each criterion. You will see excerpts from the Governance Criteria relating to the areas where the Baseline Analysis identified gaps. It will be beneficial to read the Governance Criteria in their entirety first to understand interconnections, concepts etc. At the same time, data and self-reflection produced for the Baseline Analysis will also be useful later on, during the Materiality Analysis. Keep in mind that the feedback from the Baseline Analysis is a crucial input for the process of creating the Patient Engagement Plan.
You can re-take the Baseline Analysis questionnaire at any time. This will, naturally, result in re-scoring and an update of the recommendations.
7.5 Patient Engagement Plan
This tool is essentially a digital and interactive version of the Appendix 3: Patient Engagement Plan Template.
The prerequisite for preparing the Plan is establishing the Engagement Coordination Team (ECT) as this governance body is responsible for preparing the Plan. You need to invite the ECT’s chair and other members to the Toolbox and assign them roles, so they have access and rights to the fill in the tool. It is important that the ECT’s members undergo training as outlined in the Patient Engagement Guidelines and are given enough time to reflect on this task and contact the patients if needed.
After the ECT fills in the Plan, you can download it in the .pdf format. It is also possible for members of you initiative to discuss in the comment section under the Plan.
Accomplishing the Patient Engagement Plan allows you to proceed to the Materiality Analysis.
7.6 Materiality Analysis
Identification of stakeholders
For each stakeholder category, which your initiative identified in the Baseline Analysis, your initiative needs to engage at least five representatives. This is important for ensuring a balanced voting process. At least 16 participants must take part in the survey before the tool produces final results. They do not have to register in the Toolbox to participate. For those unregistered, you need to add their e-mail addresses under corresponding stakeholder category to allow them to participate.
Creating the invitation list
After clicking the “Perform Materiality Analysis” button (1) and saving, you go the Admin Panel (2).
In the Admin Panel, click the „Edit invitation list” button. In the invitation list, write (or paste) the text of the invitation e-mail and reminder e-mail. The Toolbox will send them for you. Using “Add member” button, add e-mail addresses of your stakeholders and select their stakeholder category. Once you finish – save your work.
It is practical to prepare texts of two e-mails before launching the tool: one inviting the stakeholders to take part in the materiality analysis and one reminding them to do so. You may want to explain what materiality analysis is, why they are invited, and what is expected of them, as well as assure them of anonymity. It may be prudent to give deadlines for response. Consequently, reflect on how much time the whole process of the materiality analysis can take within your initiative’s unique timeline, and plan accordingly.
Initiation
By clicking „Email Invitation” button, you initiate the materiality analysis: an email is sent to all the participants you listed in the previous step. In addition to the text you provided, it contains a secure link to the Materiality Analysis interface in the Toolbox. Tokenization of the link guarantees:
· Anonymity of the participants.
· Association of every entry with its respective stakeholder category.
· One-vote-per-participant rule.
Selection of aspects
The participants are requested to select CRIF aspects that are relevant and significant from their perspective. They can read concise descriptions of aspects and are provided with concise instructions and links to additional material explaining CRIF and materiality analysis. Each participant must select minimum one aspect from each of the five CRIF dimensions and prioritize the aspects they selected by grading them from 0 (non-relevant) to 6 (most relevant), separately for each dimension. At least two aspects in every dimension must be graded above 0 before participant may submit their form.
The Toolbox ensures that nobody, including you will know how any individual participant voted, or whether they voted at all. You will be able to monitor how many participants from each stakeholder category voted, and track frequency of responses on a timeline. Only Promoters can initially see intermediate results and monitor level of participation. When the analysis is complete, all participants are able to see the results as.
Selection of indicators
You can see the selected aspects by clicking on “Edit Results” in the initiative overview page.
The aspects are selected by the participants, the initiative controls how many will to be used.
|
Aspect selected by the materiality analysis participants. |
|
Description of the aspect. |
 |
Indicator related to the aspect. The initiative can select which indicators related to the selected aspects it wants to use. Detailed descriptions of the indicators found in the Master Scorecard browser may help in the process. |
 |
Description of the selected indicator (it will change when you change the indicator). |
 |
Average score based on participants’ prioritisation scores. |
Final results
The Materiality analysis shows a snapshot in time of the stakeholders’ priorities; however, they may change over time. For this reason, a materiality analysis should be carried out periodically, on a yearly or biyearly basis, in order to ensure its alignment with stakeholders’ priorities and their commitment to accomplishment of the initiative’s mission and agenda. Past materiality analysis results are stored in the initiative’s database which enables comparison.
7.7 Impact Assessment Dashboard: the Master Scorecard
After you submit the final results of the materiality analysis, Assessment Dashboard section will appear in the Toolbox. Both the members of the initiative and the participants of the materiality analysis can see the initiative’s scorecard by clicking on the Final Results link. The scorecard can be downloaded as a .pdf.
7.8 Impact Assessment (PRO)
The impact assessment includes additional tool for assessing an initiative’s impact using Patient Reported Outcomes (PRO) data. The link to this tool is always available at the bottom of the initiative overview page. You can download an example file containing anonymized patient data of their periodical Hospital Anxiety and Depression Scale assessment, and upload it back to the Toolbox to see how the tool works. The Toolbox produces graphs to portray the progress of the collective number of patients or of individual ones from the uploaded data. Furthermore, you can add individual patient’s data “by hand”.
PARTNERS OF THE MULTI-ACT CONSORTIUM
The MULTI-ACT consortium brings together European societies, patients, patient organizations, research and academic institutions, and private consultancies.
|
Coordinator |
|
|
|
The Italian Multiple Sclerosis Society Foundation (FISM) is the leading funding agency of research in the field of multiple sclerosis (MS) in Italy and the third worldwide (after MS Societies in the USA and Canada). FISM is member of the International MS Federation and collaborates with other MS societies to improve the quality of life of people with MS (“PwMS”) and to provide better treatments toward a definitive cure for a MS. The overall goal of FISM is to make the bridge walkable between PwMS and governmental healthcare and research agencies, and thus to support people with MS in making decisions for their treatments and quality of life. As Coordinator of the project, FISM act as boundary organization between research and patients and society. |
|
Partners |
|
|
|
Università degli Studi di Trento, UNITN is responsible for the coordination among academic partners. The Department of Economics and Management (DEM) of the University of Trento features a multidisciplinary research environment where researchers apply a vast array of different approaches to describe the choice of economic agents, investigate their determinants and analyse their effect at the individual, sectoral and aggregate level. |
|
|
ERNST & YOUNG Italy, EY, is the partner responsible for the design and implementation of the health collaborative initiatives’ approach and policies. EY is a global leader in advisory, assurance, tax, and transaction services. The insights and quality services EY delivers help build trust and confidence in the capital markets and in economies all over the world. |
|
|
Universidad de Burgos, UBU, contributes to the MULTI-ACT Project with theoretical insights and empirical evidence about accountability, indicator measurement and impact assessment of research across different dimensions. |
|
|
Tampere University is a higher education institution with the social mission of educating visionaries who understand the world and can change it towards the better. The new multidisciplinary Tampere University brings together research and education focusing on technology, health and society. |
|
|
The European Brain Council (EBC) is a non-profit organization aiming to promote brain research in Europe, improve treatment, care and quality of life of people living with brain disorders. EBC stimulates dialogue between scientists, society and all interested parties by promoting collaboration of member organizations with the European Commission, the European Parliament and other relevant EU and international institutions. |
|
|
INTRASOFT International S.A., INTRA is a leading European IT Solutions and Services Group with strong international presence, offering innovative and added-value solutions of the highest quality to a wide range of international and national public and private organizations. It has proven expertise in conceptual system architecture and system design, advanced application development and integration services, information portal management and communication services and project management. |
|
|
European Health Management Association, EHMA, is a Belgium-based non-profit membership organisation that focuses on enhancing the capacity and capability of health management in order to deliver high quality healthcare. EHMA operates at an international, European and national level, with a membership of over 80 organisations and individuals and a broader network in excess of 5,000. Its activities revolve around three key work streams: membership-focused actions and network engagement; research and EU project work focused on dissemination and stakeholder engagement; and events and workshops. |
|
|
Fondation pour l’Aide à la recherche sur la Sclérose en plaques, ARSEP is the leading funding agency of research in the Multiple Sclerosis (MS) field in France. ARSEP, taking advantage of its international network, including the International MS Federation (MSIF) and the Progressive MS Alliance (PMSA), has a leading role in enabling patient-reporting and in communication and /dissemination of scientific results to people with Multiple Sclerosis, families, friends, and caregivers. |
|
|
Dane-i-Analizy.pl Sp. z o.o., DiA, is a company developed by Jagiellonian University academics. It focuses mainly on the health care sector, dealing with data analysis, producing analysis and reports on data presentation and innovation and providing modern solutions for public administration. |
|
|
Universidade Catolica Portuguesa, UCP, is an autonomous higher research and education institution in Portugal. The Católica Lisbon School of Business & Economics at UCP is an internationally recognized centre of research excellence in management and economics and the leading business school in Portugal since 2008. |
ACRONYMS
|
ARSEP |
Fondation Pour L'aide A La Recherche Sur La Sclérose En Plaques |
|
BA |
Baseline Analysis |
|
CC |
Compliance Committee |
|
CRIF |
Collective Research Impact Framework |
|
DiA |
Dane-i-Analizy.pl sp. z o.o. |
|
EBC |
European Brain Council |
|
EC |
European Commission |
|
ECT |
Engagement Coordination Team |
|
EHMA |
European Health Management Association |
|
EU |
European Union |
|
EY SPA |
Ernst & Young Financial Business Advisors |
|
FISM |
Fondazione Italiana Sclerosi Multipla FISM Onlus |
|
INTRA |
Intrasoft International |
|
LB |
Leadership Board |
|
MA |
Materiality Analysis |
|
MSC |
Master Scorecard |
|
MSCU |
Multiple Sclerosis Care Unit |
|
PAB |
Patient Advisory Board |
|
PE |
Patient Engagement |
|
PRD |
Patient-reported dimension |
|
PROMs |
Patient Reported Outcomes Measures |
|
PROs |
Patient Reported Outcomes |
|
PwMS |
People with multiple sclerosis |
|
R&I |
Research and Innovation |
|
RFPO |
Research Funding and Performing Organization |
|
ROE |
Return on Engagement |
|
ROI |
Return on Investment |
|
RRI |
Responsible Research & Innovation |
|
SAB |
Stakeholder Advisory Board |
|
TAU |
Tampereen Yliopisto |
|
UBU |
Universidad De Burgos |
|
UCP |
Universidade Catolica Portuguesa |
|
UNITN |
Università Degli Studi Di Trento |
|
WG |
Committees and Working Groups |
GLOSSARY
Agenda: fundamental transformative objectives agreed upon by stakeholders that an initiative aims to achieve to fulfil its mission.
Appliers: RFPOs grouped in a multi-stakeholder initiative (e.g. Alliance) who implement the CRIF.
Beneficiaries: individuals that benefit from the long-term direct or indirect effects of the initiative, which could be for example patients, their families and caregivers.
Breaking down the boundaries: see Research & Innovation Path.
Care providers: see Stakeholder.
Co-design: see Levels of Engagement.
Compliance Committee, CC: see Governance bodies.
Conduct & operate: see Research & Innovation Path.
Consult: see Levels of Engagement.
Collective Research Impact Framework, CRIF is a conceptual framework developed by MULTI-ACT enabling a new collective accountability approach to managing and assessment multi-stakeholder R&I initiatives.
CRIF Dimensions are a set of grouped indicators from Master Scorecard for assessing the impact of an initiative. For more, see CRIF Dimensions. There are five CRIF Dimensions defined below:
· Efficacy: refers to the capacity of a given initiative or programme to achieve its mission (strategic priorities set via the stakeholder engagement process). For more, see Efficacy dimension.
· Excellence: concerns the quality of research and its findings. For more, see Excellence dimension.
· Social: considers the direct and indirect effects of health research for the whole society, going beyond patient needs. For more, see Social dimension.
· Economic: refers to long-term financial sustainability of health R&I initiatives. For more, see Economic dimension.
· Patient-reported: concerns patients whose needs and perspectives must be understood and incorporated into health research impact evaluation. For more, see Patient-reported dimension (PRD).
Criteria and sub-criteria: a set of guiding principles that constitute the MULTI-ACT Governance Model and are intended to be followed by the Model's user.
Design & plan: see Research & Innovation Path.
Design and planning: see Research & Innovation Path.
Economic: see CRIF Dimensions.
Engagement Coordination Team, ECT: see Governance bodies.
Efficacy: see CRIF Dimensions.
Evaluating research: see Research & Innovation Path.
Excellence: see CRIF Dimensions.
Executing research: see Research & Innovation Path.
Experiential knowledge: knowledge gained through experience, as opposed to a priori (before experience) knowledge.
Framework: see Multi-stakeholder framework.
Governance bodies: groups with specific roles within a multi-stakeholder initiative that are composed by individuals participating to the initiative itself. For more, see Governance bodies.
· Engagement Coordination Team (ECT) is in charge of coordinating the engagement of stakeholders, including patients, relatives and caregivers, in all the operations. For more, see Engagement Coordination Team (ECT).
· Committees and Working Groups (WG) can be appointed by the LB according to the specific needs of the program/project and the activities that will be carried out in order to achieve the desired change. For more, see Committees and Working Groups (WGs).
· Compliance Committee (CC) is in charge of maintaining a balance among stakeholders’ stances and expectations and oversee the ethical issues that might arise during the implementation of the initiative. For more, see Compliance Committee (CC).
· Leadership Board, LB: is composed by representatives from the categories of stakeholders that have a strategic importance for the initiative and represents the decision-making body. For more, see Leadership Board (LB).
· Patient Advisory Board, PAB: may be a separate body or group representing patients within the Stakeholder Advisory Board (SAB). It is composed of patient representatives from the SAB. For more, see Patient Advisory Board (PAB).
· Secretariat/Management Team may be two different bodies or one. It depends on the size and structure of the multi-stakeholder initiative. It supervises administrative and operational tasks. For more, see Secretariat/Management Team.
· Stakeholder Advisory Board, SAB: a governance body composed by interested stakeholders and provides advices to the LB. Within this board, patients, their families and caregivers (one of the categories of stakeholders involved) might be asked by the LB to provide their specific contribution and advice for the most crucial decision-making processes according to the specific need of the initiative. This category of stakeholders can be defined as a sub-group within the SAB, called Patient Advisory Board (PAB). For more, see Stakeholder Advisory Board (SAB).
Governance Initiative: is a stage in multi-stakeholder initiative (including RFPOs) implementation process concerned with governance and management of a programme or a project, see Research & Innovation Path.
Health Research & Innovation, Health R&I: refers to “activities of research, technological development, demonstration and innovation, including the promotion of cooperation with non-EU countries and international organisations, the dissemination and optimisation of results and mobility of researchers in the Union (Eur-lex, no date) within the healthcare domain.
Impact: is the long-term socio-economic changes the intervention brings about (e.g. over 5 years).
Impact Indicator: is a "quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of a development actor” (OECD, 2010).
Industry: see Stakeholder.
Inform: see Levels of Engagement.
Initiative: see Multi-stakeholder initiative.
Input: the contributions made or required by each stakeholder/organization. It can include financial, human, technical and relational resources.
Involve: see Levels of Engagement.
Leadership Board, LB: see Governance bodies.
Levels of Engagement: a way of describing the varying depth of patients’ involvement and agency in the research and innovation (R&I) process. For more, see Levels of Engagement.
· Co-design: Stakeholders are engaged with a decision-making role. For more, see Levels of Engagement.
· Involve: Stakeholders participate in research design and development as co-researchers and are engaged by providing their perspective and data. They are not involved in co-designing of the project as decision-makers. For more, see Levels of Engagement.
· Consult: Stakeholders provide feedback for decision-making, give advice and opinions, but do not participate in decision-making. For more, see Levels of Engagement.
· Inform: Stakeholders are informed about research in a passive role. For more, see Levels of Engagement.
Management Team: see Governance bodies.
Mission: initiative’s current and future role, what it wants to achieve, and how it wants to achieve it.
Monitoring and evaluation framework a logical sequence that explains causal relations between inputs, processes, outputs, outcomes and impacts. It offers indicators of each of this stages and serves organizations in monitoring and evaluating their progress towards the goals and their impact.
Multi-stakeholder framework: is a conceptual structure applicable by/to a variety of stakeholders. Framework examples include (but are not limited to) guidelines, standards, certifications, normative schemes, etc.
Multi-stakeholder initiative: is a governance structure that seeks to bring different stakeholders together to participate in the dialogue, decision-making and implementation of solutions to the shared problems or goals.
Outcome: is the intermediate results and effects of the intervention (e.g. within 5 years), and is less tangible than outputs.
Output: is the activity in relation to each stakeholder’s inputs in quantitative terms. Alternatively, it can be defined as the tangible and intangible products resulting from research and innovation.
Patient Advisory Board, PAB: see Governance bodies.
Patient-Provided Information: a range of input or data that is collected from patients.
Patient-Reported Outcomes, PROs: “any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else” (FDA, 2009).
Patient-Reported Outcomes Measures, PROMs: standardized, validated questionnaires (which are also called instruments) completed by patients to measure their perception of their functional well-being and health status.
Patient-Reported Dimension, PRD: see CRIF Dimensions.
Patients’ organizations: see Stakeholder.
Patients see Stakeholder.
Payers and purchasers: see Stakeholder.
Policy makers: see Stakeholder.
Process : “includes all the activities that enable the research to happen (i.e. reviewing of evidence, data collection, analysis, reporting and so forth)” (Hinrichs-Krapels and Grant, 2016).
Program Level: see Research & Innovation Path.
Project Level: see Research & Innovation Path.
Patient Engagement: is the action of engaging patients and their communities in R&I as key stakeholders with a decision-making role, “occurring when people with and affected by the disease meaningfully and actively collaborate in the governance, priority setting, and conduct of research, as well as in summarizing, distributing, sharing, and applying its resulting knowledge” (de Wit et al., 2013).
R&I Path: see Research & Innovation Path.
R&I: see Health Research & Innovation.
Research Funding and Performing Organizations: see Stakeholder.
Research & Innovation Path (R&I Path): refers to sequence of processes and activities in the R&I where patients can be engaged in order to maximize the impact of R&I. For more, see Research & Innovation Path. Governance program level and project development levels are distinguished (also see Governance Initiative):
· Program level: steps in multi-stakeholder initiative process concerned with the governance and management of research funding & performing programs:
o Breaking down the boundaries: conditions that should be set in RFPOs in order to facilitate patient engagement as standard practice.
o Setting research priorities: actions to establish justified interest in a specific research domain to a certain higher degree, importance, precedence, or rank over others.
o Steering institutions: actions performed to establish steering and advisory committees and bodies.
o Design and planning: the design and planning of all the activities that lead to the realization of a concept or idea and which helps achieve the item's designated objective(s).
o Executing research: activities to actualize the research program or a specific research project for the purpose of achieving the item's designated objectives. Project level takes places at this stage.
o Evaluating research: activities to determine the value created by a research program or project, establishing the outputs and outcomes, the degree to which the pre-established goals were achieved, and their impact.
o Translation to community: activities to foster and facilitate the uptake of results of research programs or projects.
· Project level: steps in multi-stakeholder initiative process concerned with performing single research projects. In this case, patient is a co-researcher. Project development pertains to research execution stage of the governance program level.
o Conduct & operate: project conduct & monitoring (e.g. ICT device development).
o Design & plan: the design and planning of all the activities that lead to the realization of a concept or idea and which helps achieve the designated objective(s).
o Evaluation activities to determine the value created by a research project, establishing the outputs and outcomes, the degree to which the pre-established goals were achieved, and the impact.
o Translation to community: activities to foster and facilitate the utilization/uptake of results of research projects.
Responsible Research and Innovation, RRI: research and innovation process in which societal actors work together in order to better align its outcomes with the values, needs and expectations of society (European Commission, no date).
Return on Engagement, RoE: the benefit, impact and value resulting from performing engagement in R&I. For more, see Return on Engagement (RoE).
Return on Investment, ROI: is a measure of the efficiency of an investment as a percentage of return relative to the investment’s cost.
RRI: see Responsible Research and Innovation.
Science of patient input: scientific research which uses data provided by people with a disease through passive or active contribution to evaluate its impact. For more, see Science of/with patient input.
Science with patient input: scientific research where patients actively collaborate in governance, setting priorities, performance assessment etc. For more, see Science of/with patient input.
Setting research priorities: see Research & Innovation Path.
Social Return on Investment, SROI: a method of measuring and accounting for extra-financial value (such as environmental or social value) for the stakeholders.
Social Dimension: see CRIF Dimensions.
Society: see Stakeholder.
Stakeholder: refers to “any individual or group that is affected by, who can influence or may have an interest in the outcomes of an organization’s actions” (Freeman, 1984).
· Patients: people with the disease (persons with lived experience of the disease); and people affected by the disease (persons or groups that are affected by the disease, including family members and caregivers).
· Patients’ organizations: not-for profit organisations which are patient focused, where patients and/or their carers constitute majority in governing bodies, e.g. patient associations, advocacy organizations.
· Society: individual citizens, civil society organizations and networks.
· Payers and purchasers: public or private entities responsible for underwriting the costs of health care.
· Care providers: health and social care organizations and professionals (doctors, nurses, etc.).
· Policy makers: EU institutions; national, regional and local policy makers of different levels.
· Regulators: regulatory agencies and Health Technology Assessment (HTA) bodies, e.g. agencies for the scientific evaluation and safety monitoring of medicines, i.e. the European Medicine Agency.
· Industry: companies developing and selling health products and services.
· Research Funding and Performing Organizations, RFPO: universities, research hospitals, research projects, foundations, and all private and public research funders.
Promoter: individuals that guide the adoption of the CRIF within their organizations or initiatives, and are members of their governance bodies.
Stakeholder Advisory Board, SAB: see Governance bodies.
Steering institutions: see Research & Innovation Path.
Sub-criterion: see Criteria.
Transformational mission: mission of research initiative that shifts or breaks existing scientific paradigms.
Translation to community: see Research & Innovation Path.
BIBLIOGRAPHY
Andreaus, M. et al. (2019) ‘Deliverable 3.6 Multi-Act Master Scorecard’. Available at: https://www.multiact.eu/wp-content/uploads/2020/02/D3.6.pdf.
Andreaus, M. and Costa, E. (2014) ‘Toward an Integrated Accountability Model for Nonprofit Organizations’, in, pp. 153–176. doi: 10.1108/S1041-706020140000017006.
Cabaj, M. and Weaver, L. (2016) Collective Impact 3.0: An evolving framework for community change. Available at: https://www.tamarackcommunity.ca/library/collective-impact-3.0-an-evolving-framework-for-community-change.
Cottam, H. and Leadbeater, C. (2004) Red Paper 01 Health: Co-creatingServices. Available at: https://www.designcouncil.org.uk/sites/default/files/asset/document/red-paper-health.pdf.
Department of Health (2009) ‘Guidance on the routine collection of Patient Reported Outcome Measures (PROMs)’, (April 2008), pp. 1–28. Available at: http://www.dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/documents/digitalasset/dh_092625.pdf.
Deuschl, G. et al. (2020) ‘The burden of neurological diseases in Europe: an analysis for the Global Burden of Disease Study 2017’, The Lancet Public Health, 5(10), pp. e551–e567. doi: 10.1016/S2468-2667(20)30190-0.
Dhar, V. K. et al. (2018) ‘Benefit of social media on patient engagement and satisfaction: Results of a 9-month, qualitative pilot study using Facebook’, Surgery, 163(3), pp. 565–570. doi: 10.1016/j.surg.2017.09.056.
Doria, N. et al. (2018) ‘Sharpening the focus: differentiating between focus groups for patient engagement vs. qualitative research’, Research Involvement and Engagement, 4(1), p. 19. doi: 10.1186/s40900-018-0102-6.
Eur-lex (no date) Glossary of summaries. Available at: https://eur-lex.europa.eu/summary/glossary/research_and_development.html.
European Brain Council (2017) Early Intervention: Bridging the Early Diagnosis and Treatment Gap. doi: 10.13140/RG.2.2.20145.07520.
European Commission (no date) Horizon 2020. Available at: https://ec.europa.eu/programmes/horizon2020/en/h2020-section/responsible-research-innovation.
European Medicines Agency (2014) EMEA Appendix 2 to the guideline on the evaluation of anticancer medicinal products in man.
FDA, H. (2009) ‘Guidance for Industry Patient-reported Outcome measures: Use in Medical Product Development to Support Labeling Claims’, Clinical/Medical Federal Register, (December), pp. 1–39. doi: 10.1111/j.1524-4733.2009.00609.x.
Fontaine, G. et al. (2019) ‘Communicating Science in the Digital and Social Media Ecosystem: Scoping Review and Typology of Strategies Used by Health Scientists’, JMIR Public Health and Surveillance, 5(3), p. e14447. doi: 10.2196/14447.
Freeman, E. R. (1984) Strategic management : a stakeholder approach. Latest edi. Boston, MA.
Graham, K. E. R. et al. (2012) ‘Evaluating health research impact: Development and implementation of the Alberta innovates-health solutions impact framework’, Research Evaluation, 21(5), pp. 354–367. doi: 10.1093/reseval/rvs027.
Gustavsson, A. et al. (2011) ‘Cost of disorders of the brain in Europe 2010’, European Neuropsychopharmacology, 21(10), pp. 718–779. doi: 10.1016/j.euroneuro.2011.08.008.
Hinrichs-Krapels, S. and Grant, J. (2016) ‘Exploring the effectiveness, efficiency and equity (3e’s) of research and research impact assessment’, Palgrave Communications, 2(1), p. 16090. doi: 10.1057/palcomms.2016.90.
Househ, M., Borycki, E. and Kushniruk, A. (2014) ‘Empowering patients through social media: The benefits and challenges’, Health Informatics Journal, 20(1), pp. 50–58. doi: 10.1177/1460458213476969.
Jeremy Nicholls et al. (2012) ‘Social Return A guide to on Investment Social Return on Investment’, Update, (January), p. 108. Available at: http://www.thesroinetwork.org/publications/doc_details/241-a-guide-to-social-return-on-investment-2012.
Kalucy, E. C. et al. (2009) ‘The feasibility of determining the impact of primary health care research projects using the Payback Framework’, Health Research Policy and Systems, 7, pp. 1–10. doi: 10.1186/1478-4505-7-11.
Kania, J. and Kramer, M. (2011) ‘Collective Impact’, Stanford Social Innovation Review, Winter. Available at: https://ssir.org/images/articles/2011_WI_Feature_Kania.pdf.
Mazzucato, M. (2018) Mission-Oriented Research & Innovation in the European Union. A problem-solving approach to fuel innovation-led grow. doi: 10.2777/36032.
Milat, A. J., Bauman, A. E. and Redman, S. (2015) ‘A narrative review of research impact assessment models and methods’, Health Research Policy and Systems, 13(1), pp. 1–7. doi: 10.1186/s12961-015-0003-1.
Multi-Act Project (2020) Guidelines for Patient Engagement in Health Research & Innovation. Short version Version 0.1 – World Multiple Sclerosis Day 2020, May 30th. Available at: https://www.multiact.eu/wp-content/uploads/2020/06/MULTI-ACT-Patient-Engagement-Guidelines-Short-v0.1_compressed.pdf.
Musso, M. et al. (2018) ‘How Social Media Platform can Support Value Cocreation Activities in Healthcare’, in Excellence in Services, pp. 535–555.
OECD (2010) ‘Glossary of Evaluation and Results Based Management (RBM) Terms’. Available at: https://www.oecd.org/dac/evaluation/dcdndep/31950400.pdf.
Ovseiko, P. V., Oancea, A. and Buchan, A. M. (2012) ‘Assessing research impact in academic clinical medicine: A study using Research Excellence Framework pilot impact indicators’, BMC Health Services Research, 12(1). doi: 10.1186/1472-6963-12-478.
Raftery, J. et al. (2016) ‘Models and applications for measuring the impact of health research: Update of a systematic review for the health technology assessment programme’, Health Technology Assessment, 20(76). doi: 10.3310/hta20760.
Schneeman, K., Barton, V. and Huneycutt, B. (2019) Advancing Models of Patient Engagement: Patient Organizations as Research and Data Partners. Available at: https://milkeninstitute.org/sites/default/files/reports-pdf/Full Series_Advancing Models of Patient Engagement- Patient Organizations as Research and Data Partners copy.pdf.
von Schomberg, R. (2013) ‘A Vision of Responsible Research and Innovation’, in Responsible Innovation. Chichester, UK: John Wiley & Sons, Ltd, pp. 51–74. doi: 10.1002/9781118551424.ch3.
Strand, R. et al. (2015) Indicators for Promoting and monitoring responsible research and innovation. doi: 10.2777/9742.
Thielst, C. B. (2011) ‘Social media: ubiquitous community and patient engagement.’, Frontiers of health services management, 28(2), pp. 3–14. Available at: http://www.ncbi.nlm.nih.gov/pubmed/22256506.
WHO (2014) Handbook for Guideline Development. 2nd edition. Available at: https://www.who.int/publications/guidelines/handbook_2nd_ed.pdf?ua=1.
de Wit, M. P. T. et al. (2013) ‘What Has Been the Effect on Trial Outcome Assessments of a Decade of Patient Participation in OMERACT?’, The Journal of Rheumatology, 41(1), pp. 177–184. doi: 10.3899/jrheum.130816.
Wittchen, H. U. et al. (2011) ‘The size and burden of mental disorders and other disorders of the brain in Europe 2010’, European Neuropsychopharmacology, 21(9), pp. 655–679. doi: 10.1016/j.euroneuro.2011.07.018.
Yaghmaei, E. (2018) ‘Responsible research and innovation key performance indicators in industry: A case study in the ICT domain’, Journal of Information, Communication and Ethics in Society, 16(2), pp. 214–234. doi: 10.1108/JICES-11-2017-0066.
Zaratin, P. et al. (2016) ‘Progressive MS Alliance Industry Forum: Maximizing Collective Impact To Enable Drug Development’, Trends in Pharmacological Sciences, 37(10), pp. 808–810. doi: 10.1016/j.tips.2016.07.005.
Zaratin, P., Battaglia, M. A. and Abbracchio, M. P. (2014) ‘Nonprofit foundations spur translational research’, Trends in Pharmacological Sciences, 35(11), pp. 552–555. doi: 10.1016/j.tips.2014.09.003.
[1] Goal is a description of a destination, and an objective is a measure of the progress that is needed to get to the destination.
[2] The products, capital goods, and services that result from a development intervention.
[3] An initiative should have 100% accountability of these two elements for which it is considered accountable. Differently, the transformative objective in most cases is affected by external factors and variables – this obviously reduces the initiative’s accountability over the effective achievement of the results.