Figure 6 The MULTI-ACT CRIF Workflow and the relation with the Co-accountability Pillars

 

 

Phase 1

Three steps of the phase 1 lead to the definition of your initiative’s mission. The mission usually remains unchanged in the long run:

·        If your initiative is already set up, conduct baseline analysis in order to measure its level of compliance with the MULTI-ACT Governance Model.

·        Your initiative identifies its intended beneficiaries, analyzes its operating context, and learns about needs of its stakeholders. If the “patients” stakeholder category is selected, then a patient engagement plan should be defined (see sub-criterion 2.1).

·        On this basis, your initiative defines its new mission or refines an existing one.

Phase 2

Through phase 2, MULTI-ACT proposes a specific methodology for defining the material topics which establish the agenda of the initiative: the collective materiality analysis.

Next phase is collective materiality analysis. It is a way for your initiative to engage its stakeholders in defining which topics are significant and relevant for them. Based on that, the initiative can define next steps towards meeting their expectations.

Phase 3

Based on the material topics selected through the materiality analysis, your initiative can outline its agenda, identifying the transformative objectives that reflect the stakeholders’ perspective.

Phase 4

The agenda needs be monitored through a measurement system: relevant indicators associated with the material aspects in the MULTI-ACT Master Scorecard. Once the indicators associated with the relevant aspects are identified, the initiative should put in place a consistent and efficient data collection procedure, in order to gather effectively and on a regular basis, the requested information .

Phase 5

You are encouraged to use the Toolbox to conduct the co-selection process of the aspects and indicators that best reflect the issues relevant for the stakeholders. The Scorecard contains 12-15 aspects chosen from a list of 53, and 12-15 indicators chosen from the among 125. that the model makes available in its impact assessment scorecard.

The circle closes with the publication of the periodic report of the initiative, which MULTI-ACT suggests to produce annually, which provides the basis for the analysis of the differences between what was planned and what was achieved, allowing to identify the appropriate improvements of the agenda of the initiative. Indeed, for a mission-oriented approach, while the mission is defined at the beginning of the initiative, the alignment of the agenda to the mission needs to be monitored and checked regularly, and therefore, phases 2 to 5 should be repeated accordingly (e.g. on an annual basis). Your initiative needs to base the entire process (phases 1 to 5) and application of the Tools on continuous engagement of its stakeholders, especially patients. Patient Engagement Guidelines will help you do it correctly.

Governance Model: Governance Criteria and Patient Engagement

Understanding governance and patient engagement: important concepts

Below you will find explained some concepts which are crucial for this chapter. They are arranged in thematic groups rather than in alphabetical order to facilitate comprehension.

Mission

Mission defines your initiative’s current and future role, what it wants to achieve, and how it wants to achieve it.

Agenda

Agenda is a list of fundamental transformative objectives agreed upon by stakeholders that an initiative aims to achieve to fulfil its mission, including a description of the main outputs and activities needed to achieve them.

Stakeholder classification

Stakeholder is an individual or group that is affected by the outcomes of your initiatives’ actions, or who can influence these outcomes or may have an interest in them[6]. In other words, stakeholders are people, communities, organisations and other entities that experience a change – positive, negative or neither – as a result of the activities of your initiative. Some of them will be participants of your initiative, others will not be even aware of its existence. Using a stakeholder classification is a useful way of thinking about people and organizations relevant to you initiative. While not all of them are equally important, nor are they all going to be involved to the same degree, it is important not to overlook any group influenced by the initiative. You can find the whole general stakeholder classification in the glossary, and below explanations of the categories that are less self-evident meanings[NM2] :

• Patients are people with disease and people affected by the disease (i.e. relatives, caregivers). It is important to keep in mind that the term “patients”, as used throughout this Manual, includes family, significant others, and caregivers of persons with the disease. This is in recognition of the fact that all these people may provide crucial information about influence of your initiative on lives of persons with the disease and those around them.
• Care providers are health and social care organizations and professionals (doctors, nurses, assistants etc.). Caregivers and care providers should not be confused. In this Manual, “caregivers” do not provide their care in a professional capacity, unlike care providers.
• Research Funding and Performing Organizations (RFPOs) are universities, research hospitals, research projects, foundations, and all private and public research funders. Most of organizations participating in your initiative will likely fall into this category. It is worth remembering that RFPOs can differ widely one from another, so the communication between them may be challenging, not to mention different motivations and goals.

Additionally, the CRIF Manual refers to promoters, appliers and beneficiaries of the CRIF, when describing how stakeholder organizations and their representatives participate in governance of your initiative.

• Promoters are individuals that guide the adoption of MULTI-ACT within their organizations or initiatives. They can be either already existing multi-stakeholder organizations or initiatives, with an defined governance structure or newly-established ones, willing to fully adopt MULTI-ACT governance approach. They represent various stakeholder categories, most often RFPOs, industry and Patient organizations.
• Appliers are RFPOs grouped in a multi-stakeholder initiative who implement the MULTI-ACT Framework.
• Beneficiaries are individuals benefitting in the long-term, directly or indirectly, from a multi-stakeholder initiative. Particular focus is on Patients, Patients organizations and society.

Governance bodies

Governance bodies are groups with specific roles within a multi-stakeholder initiative that are composed of individuals participating to the initiative itself. The suggested governance bodies to be established are presented in the figure below.

 

Figure 7 Governance Bodies

 

Leadership Board (LB)

Function	LB is the decision-making body within the governance structure (recommendation 2.3.1). It oversees fulfilment of the mission and agenda, and coordination and implementation of activities of your initiative. It supervises Working Groups, Committees, and administration (recommendation 2.3.2). It enforces deadlines and improve your initiative’s performance (sub-criterion 4.2), with help from the Management Team if needed. The LB evaluates and chooses actions and tools (e.g. Progress Report, to respond to current needs of the beneficiaries and changing circumstances (sub-criterion 4.3). It delegates tasks to WGs or other bodies as needed. It leads the review process (recommendation 5.1.7), with the SAB. LB creates a procedure formalizing various aspects of how the initiative functions, from governance bodies appointment to stakeholder engagement (recommendation 2.3.3), with support of CC and the ECT. This procedure needs to be approved by the SAB. LB appoints (recommendation 2.3.2): Chair/coordinator acting as an internal and external point of reference for the initiative; Operational teams, such as a sub-board and the Secretariat/ Management Team – when needed Working Groups, Committees, and collaborative team to carry out various tasks LB is responsible for formalizing procedures and strategies: It creates the Engagement Plan (sub-criterion 3.1, Implement phase) with ECT and SAB. LB has a responsibility to define the collective Action Plan and enforce its implementation (sub-criterion 4.1) through establishment of dedicated WGs, and creating accountability mechanisms. It creates a process for collecting feedback, opinions, and grievances of internal and external stakeholders (recommendation 5.2.1), with the ECT. It formalizes procedures on how the initiative’s participants interact with each other, balancing stakeholder engagement and agile management (sub-criterion 3.4). with the ECT and the CC. The LB also determines the budget and conduct a cost analysis of the initiative, as well as identifies critical issues and gaps in your initiative’s operations (sub-criterion 4.4). It may delegates these tasks to the Secretariat/Management Team. LB Identifies gaps in stakeholder engagement capacity (sub-criterion 3.1) and then monitors, evaluates and improves quality of stakeholder engagement (sub-criterion 3.1), with SAB. It ensures appropriate communication to relevant stakeholders and the general public (recommendation 5.3.1) The LB is responsible for constantly maintaining an alignment between the shared assessment system and the mission and agenda of the initiative (recommendations 5.1.4 and 5.1.5).
Appointment	LB is set up by the Promoters. Composition of the LB needs to be approved by the SAB and PAB (sub-criterion 2.3).
Composition	The composition of the LB reflects categories of the stakeholders that participate in your initiative and have strategic importance. Its members act as these categories’ representatives. Their number varies according to the initiative’s needs. LB has to be balanced in terms of gender, sector and geographical background, language, political diversity, perspectives and experiences. The members of LB should be committed and skilled individuals, which should ensure constant participation to the initiative’s development. LB members hold equal power, because it guarantees equity among participant stakeholders. The composition of the LB and its members should undergo the approval of the SAB and the PAB (recommendation 2.3.2).

Table 4 Leadership Board (LB)

Engagement Coordination Team (ECT)

Function	The ECT coordinates the involvement of stakeholders, including patients, in all the operations. It coordinates all training and coaching activities to facilitate the stakeholders’ engagement (sub-criterion 3.2), which includes providing briefing materials and organizing training sessions. Cooperation between the ECT and the LB plays a crucial role in the initiative’s governance. While the LB provides agile management, the ECT should guarantee and facilitate the participation of weak and/or marginalized stakeholders as well as a balance among different points of view (sub-criterion 3.4). The ECT works as a facilitator between the SAB and the LB (recommendation 2.1.1). It identifies gaps in stakeholder engagement capacity (sub-criterion 3.1), together with the LB. It assesses stakeholder engagement (sub-criterion 3.1), together with the LB and the SAB. It ECT maintains the active participation of the internal stakeholders in the LB-led process of setting up a stakeholder feedback mechanism (sub-criterion 5.2). In terms of patient engagement responsibilities (Composition and skills of Engagement Coordination Team) It designs, implements and keeps the Patient Engagement Plan up to date. Oversees translation of patients’ experiential knowledge into patient-reported outcomes. Moderates interdisciplinary dialogue Translates technical language into a language that patients easily understand Mitigates issues like ethical conflicts in protocol design, tokenism, patient recruitment etc.
Appointment	Promoters establish the ECT: they recruit and appoint its members. The agreement of the LB is needed (sub-criterion 2.3, recommendation 2.1.1). You can find detailed instruction Establish the Engagement Coordination Team
Composition	The essential figures of this team are: Co-Chair, patient (1 person) Co-Chair, MULTI-ACT trained representative (1 person): this individual has to complete the MULTI-ACT Training Academy® course (see Training of Engagement Coordination Team members). Initiative’s board representative (1 person) Initiative’s staff representative (1 person) Patients (with consideration to the balance of gender, geography, disease progression, age, socioeconomic background) (3-6 persons) Expert on the mission/priorities of the initiative, e.g. Working Group representative, industry forum representative. However, the composition of this team can vary depending on the specificity of individual programs and projects. All recruited experts are encouraged to take an educational module MULTI-ACT Training Academy®.

Table 5 Engagement Coordination Team (ECT)

Stakeholder Advisory Board (SAB)

Function	The main function of the SAB is advisory – it supports the LB. It may, however, evolve over time into a decision-making body, acting like a Stakeholder Assembly. The SAB leads the review process (recommendation 5.1.7), with the LB. It confirms appointment of the CC, with the LB (sub-criterion 2.4). Patients, as a specific stakeholder category included in the SAB, may be asked by the LB for their own contribution. This group may form a sub-board of SAB: the Patient Advisory Board (PAB) (recommendations 2.1.1, 2.3.1). It approves the composition of the LB, with the PAB (sub-criterion 2.3).
Appointment	Appointed by Promoters with the contribution of the CC and the ECT. (recommendation 2.3.1).
Composition	The SAB is composed of interested stakeholders. The Promoters, with the ECT, arrange an open call for participation in the SAB. The CC and the ECT establish the rules regarding selection, composition, and balance of the SAB. PAB is a sub-board of SAB (recommendation 2.3.1).

Table 6 Stakeholder Advisory Board (SAB)

Patient Advisory Board (PAB)

Function	PAB may a separate body or group representing patients within the SAB. It presents the voice and opinions of patients, including underrepresented patients (recommendation 2.1.1). It is to be consulted by the ECT and the LB. It approves the composition of the LB, with the SAB (sub-criterion 2.3).
Appointment	Promoters with the Compliance Committee (CC) and ECT appoint PAB during creation of the SAB (sub-criterion 2.3).
Composition	PAB is composed of patient representative from the SAB (recommendation 2.1.1).

Table 7 Patient Advisory Board (PAB)

Compliance Committee (CC)

Function	CC maintains a balance among stakeholders’ stances and expectations. It oversees ethical issues too (sub-criterion 2.4). The CC takes part in the decision-making process of your initiative. It contributes to the LB’s activities, especially: Guaranteeing equity (sub-criterion 2.4) Ensuring that the self-interest of stakeholders does not prevail on collective decision-making processes (sub-criterion 2.4) avoiding tokenism (sub-criterion 2.4) Making sure that the decision-making process considers different views (sub-criterion 2.4) Managing conflicts (sub-criterion 2.4) Guaranteeing ethical acceptability and social justice of the initiatives’ objectives and activities (sub-criterion 1.4); Ensuring a balance between effective engagement of participants and agile management of the initiative (sub-criterion 3.4) Supporting the LB in formalizing a procedure (recommendation 2.3.3) It also may support Secretariat/Management Team if needed in its duties related to financial oversight (sub-criterion 4.4)
Appointment	First appointed by the Promoters in the beginning of Governance Model implementation, later officially confirmed by the LB and the SAB (sub-criterion 2.4).
Composition	It can be a committee or an individual, depending on the size, level of development and resources of your initiative (sub-criterion 2.4).

 

Committees and Working Groups (WGs)

Function	Creation of WGs is optional, and their responsibilities, role and specific tasks are assigned according to current needs of your initiative. For example, the LB may charge the bodies with research or reporting. They may also be responsible for maintaining feedback mechanism and communication (recommendation 5.3.1). WGs may carry out operative tasks, while Committees may provide insights and opinions (recommendation 2.3.1). They report to the LB and are supervised by the Management Team.
Appointment	LB appoints WGs if needed and as needed (sub-criterion 2.3).
Composition	The WGs should be composed and balanced in terms of the stakeholders’ categories and needs of the initiative (recommendation 4.1.1).

Table 8 Committees and Working Groups (WGs)

Secretariat/Management Team

Funtions	The Secretariat and Management Team may be two different bodies or one. It depends on the size and structure of the multi-stakeholder initiative. Secretariat/Management Team supervises administrative and operational tasks. The body: Enforces the deadlines and overseeing activities in your initiative (sub-criterion 4.2) Supervises of the general performance of the initiative according to the defined mission and agenda (sub-criterion 4.2). Helps the LB with the Progress Report (recommendation 5.1.6) Runs administrative duties ensures of financial security and legal compliance of your initiative (sub-criterion 4.4) oversees WGs’ activities supports the LB in gathering data for Progress Report (recommendation 5.1.6) may support the LB and the SAB in the review process (recommendation 5.1.7). supports the LB in setting up a process for gathering stakeholders’ feedback (recommendation 5.2.1). The LB may decide to delegate to the Secretariat/Management Team the tasks of determining the budget and conducting a cost analysis of the initiative, as well as identifying critical issues and gaps in your initiative’s operations (sub-criterion 4.4).
Appointment	LB appoints it/them based on the initiative’s needs and tasks to be performed (sub-criterion 4.2).
Composition	LB can decide on the composition of these bodies (or body). A multi-stakeholder approach is not required here.

Table 9 Secretariat/Management Team

Experiential knowledge

Experiential knowledge is knowledge gained through experience, as opposed to a priori (before experience) knowledge. It arises when patients’ experiences are converted – consciously or unconsciously – into personal insights that help the patient to cope with the illness [7]. When patients share their experiential knowledge, the collective experiential knowledge exceeds the sum of individual experiences.

Levels of Engagement

Stakeholders can contribute to the health research and innovation simply by getting informed or by participating in your initiative with various levels of decision-making power. Levels of Engagement presented below are a useful way of describing the varying depth of patients’ involvement with research and innovation (R&I) process. The same stakeholder may be engaged at different levels depending on the phase of the initiative, their role, or other factors. Levels of Engagement are relevant to sub-criterion 3.3 and explained in detail in Patient Engagement Guidelines.

Co-design	Stakeholders are engaged since the very beginning of the R&I initiative with a decision-making role. Patients are asked to co-define the share agenda and co-design research governance. Stakeholders are members of the Leadership Board
Involve	Patients are engaged in the research project and given an active role: they provide their perspective and data on a specific topic. However, the project is designed and initiated by professionals and patients who are not engaged in the co-design of the project as direct decision-makers. Gathering patients’ views on the topics that are important for them. Co-creation of the patient-reported outcome measurements for clinical trials development. Stakeholders act as members of the Leadership Board or Working Groups.
Consult	Stakeholders are asked to provide feedback for decision-makers about their analysis or decisions. Stakeholders participate by being asked for advice and opinion, by expressing their views and having discussions. It does not usually include any share in decision-making. Consulting activities, survey, interviews, establishing and maintaining relationship with stakeholders. Stakeholders act as members of the Stakeholder Advisory Board.
Inform	Stakeholders are informed about research priorities, activities, outcomes and impact. Patients receive information from researchers in a passive way.

Table 10 Levels of Engagement

Research & Innovation Path

Research & Innovation Path (R&I Path) – presented in a simplified manner in the figure below – is a sequence of processes and activities in the R&I, in which patients can be engaged in order to maximize the impact of R&I. The steps of the R&I Path represent stages in research performance and management of funding within the multi-stakeholder health research initiatives or individual RFPOs.

They differ slightly for the Governance Program Level, which concerns often complex research programs, and for Project Development Level, which concerns single research projects.

 

Figure 8 Research and Innovation Path

You will encounter R&I Path when undergoing the Baseline Analysis, in the recommendations 3.2.2 and 4.1.1 of the Governance Criteria, and you will find a detailed explanation of the steps on both levels in the Patient engagement in the Research & Innovation Path sub-chapter of the Patient Engagement Guidelines.

Return on Engagement (RoE)

You can think about it as akin to return on investment, but your investment is not purely monetary: it is investment in engaging patients in R&I and building relationship with them. The return is considered in broader sense – it is about various impacts and benefits resulting from performing patient engagement in your R&I initiative. RoE is discussed at length in Patient Engagement Guidelines, particularly the metrics developed to evaluate whether engagement adds value for different stakeholder groups.

Science of/with patient input

In this Manual, you will encounter terms “science of patient input” and “science with patient input”. Science of patient input is about using data provided by people with a disease through passive or active contribution to evaluate impact of R&I. You may think of it as more “traditional” way of doing research, where patients “are studied”. For example, in the context of MULTI-ACT, data about patients’ experiences [8] outside the clinic are critical to evaluate impact of mission-oriented health research on outcomes that matter most to patients. Science with patient input occurs when patients actively collaborate in the governance, setting priorities, research performance assessment etc. of R&I. You may think of them as “co-researchers”. It aims to maximize impact of R&I. The concept is relevant to the recommendation 3.2.2 and to Patient Engagement Plan and is explained in details in Patient Engagement Guidelines,.

Patient Reported Outcome (PRO)

PRO is any report about patient’s health status coming directly from the patient, and was not interpreted by anyone else [9]. PROs are strictly about patient’s perception of disease and treatment [10], so they hold a special place within CRIF, where the patient is the key stakeholder, contributing their experiential knowledge to the research.

Patient Reported Outcomes Measures (PROMs)

standardized, validated questionnaires (which are also called instruments) completed by patients to measure their perception of their functional well-being and health status (National Health Service, 2009)[NM3] . PROMs are questionnaires measuring the patients’ views of their health status. PROMs are used to assess a patient’s health status at a particular point in time. PROMs tools can be completed either during an illness or while treating a health condition. In some cases, using pre- and post-event PROMs can help measure the impact of an intervention. PROMs are tools used to measure patient-reported outcomes (PROs). PROMs are offered in the Toolbox. Their results are translated into PRD indicators.

Multi-Act Governance Model: Governance Criteria
Criterion 3: Clear, effective and inclusive methodology of stakeholder engagement
The Appliers of the Multi-Act Governance Model guarantee a comprehensive, balanced and efficient stakeholder engagement process, ensuring participation of patients and of other relevant stakeholders.
·        3.1: Define and approve a methodology to engage stakeholders ·        3.2: Engage intended beneficiaries ·        3.3: Differentiate the level of engagement according to involved stakeholders ·        3.4: Ensure a balance between engagement of involved stakeholders and agile management of the initiative
Criterion 1: Mission and agenda	Criterion 2: Participatory Governance	Criterion 4: Effective and efficient management and coordination of the initiative	Criterion 5: Co-accountability assessment
Appliers define a mission and a shared agenda, considering Multi-Act principles.	Appliers guarantee an inclusive and equitable governance model, which allows involvement of all relevant parties through a co-design approach.	Appliers guarantee an effective, cooperative and efficient coordination of the objectives and actions required to pursue the Mission and the Agenda.	Appliers establish a shared and effective measurement system, comprising of a set of indicators, which promotes continuous improvement and communication. The set a mechanism to receive feedbacks.
·         1.1: Identify intended beneficiaries, analyze the operating context of the initiative and understand the needs of stakeholders ·        1.2: Define a shared mission and common agenda ·        1.3: Promote a movement building approach to achieve transformative changes ·        1.4: Guarantee ethical acceptability and social justice	·        2.1: Allow the involvement of intended beneficiaries ·        2.2: Adopt a multi-stakeholder approach enabling co-creation ·        2.3: Implement a participatory structure ·        2.4: Guarantee equity and mechanisms to avoid self-interest	·        4.1: Enable involved stakeholders to coordinate their efforts and perform activities ·        4.2: Set clear and transparent processes and timeline ·        4.3: Maintain flexibility ·        4.4: Ensure the presence of secure funding, solid organizational structure and resources management	·        5.1: Define a shared assessment system ·         5.2: Set effective feedback mechanism ·         5.3: Ensure continuous learning, communication and disclosure of knowledge

Table 11 Multi-Act Governance Model: Governance Criteria

Baseline Analysis

Baseline Analysis is a questionnaire that measures the level of compliance [NM4] of your initiative’s governance and patient engagement with the MULTI-ACT Framework. It is recommended that you conduct the Baseline Analysis as soon as you decide to implement the MULTI-ACT Framework within your initiative. Learning the results of the Analysis has benefits regardless of how advanced the initiative is.

In the proposed model, you can perform the Baseline Analysis via the Toolbox. It contains two sets of questions which evaluate your initiative’s compliance with:

·        Five governance criteria (covered in Governance Criteria.

·        Existing practices and techniques used in patient engagement (science with patient input, covered in Patient Engagement Guidelines)

The Baseline Analysis tool will automatically provide customized governance recommendations based on the Governance Model Guidelines and Patient Engagement Guidelines, indicating gaps to be addressed. After learning the results, you will know which aspects of your initiative’s governance and patient engagement systems need development and/or correction.

During the process of filling in the Baseline Analysis questionnaire, you will be asked to upload various documents: financial reports, yearly reports, sustainability reports, internal policies on patient engagement, mission and vision, ethical compliance, monitoring and evaluation, social and environmental impact assessment, governance bodies and management procedures, academic achievement etc. While it may require effort to collect the documents, it will pay off as the Baseline Analysis results will help you identify gaps in governance of your initiative and align different procedures with the mission. You will need to come back to some of these documents when filling in the Master Scorecard. Toolbox will ask you to categorize the documents you will upload and sources you will refer to. You may find Appendix Documents classification helpful.

When you fill in the questionnaire, you will receive your score and accompanying recommendations. The score may be a pass or non-pass. Achieving a passing score means that the initiative is compliant with the Multi-Act Framework. However, you can start creating your Patient Engagement Plan in the Toolbox even if you have not yet achievement the passing score.

You can learn your compliance status for each criterion. You will see that excerpts from the Governance Guidelines relating to the areas where the Baseline Analysis found gaps. Apart from the general content and guidelines associated with every criterion, additional dynamic content is added based on the answers given to individual BA questions. The final guideline content may also include downloadable documents,

links to external sites etc. Keep in mind that the feedback from the Baseline Analysis is a crucial input for the process of creating the Patient Engagement Plan.

After getting recommendations, you can move to the subsequent chapter Governance Chapter.

In the Governance Criteria and Patient Engagement Guidelines, you will find practical procedures for implementation of governance and patient engagement guidelines. The Toolbox will direct you to the parts of the Governance and Patient Engagement Guidelines relevant for the identified gaps, but it will be beneficial to read these guidelines in their entirety to understand interconnections, concepts etc. At the same time, data and self-reflection produced for the Baseline Analysis will also be useful later on, during Materiality Analysis.

You can re-take the Baseline Analysis questionnaire at any time. This will, naturally, result in re-scoring and an update of the recommendations.

Governance Criteria

Below you will find the full text of the five Governance Criteria. We encourage you to read them in full at least once, so you will have some understanding of the whole concept. There is a lot of cross-references in the text, which are intended to facilitate your understanding of the complexities of the model.

How to use the Governance Model?

Check your initiative’s compliance with the Governance Model (both the Criteria and the Patient Engagement) through Baseline Analysis, and then focus on the areas identified as gaps. Your initiative then may focus on implementing these specific recommendations in order to become compliant with the Model. However, we recommend that you familiarize yourself with all the Criteria and recommendations, as they are inter-connected. Implementation of the Criteria may seem like a daunting task in the start, but the Model leaves your initiative as much discretion as possible, so that you can implement the recommendations in a way most suitable for your specific circumstances and mission.

Criterion 1: Mission and agenda

In the process of formulating a mission and a shared agenda for your initiative, it is important that Appliers:

•       Identify the initiative’s intended beneficiaries and analyze the context in which it operates;

•       Define a shared mission and common agenda;

•       Promote a movement-building approach to achieve transformative changes;

•       Guarantee ethical acceptability and social justice.

Sub-criterion 1.1: Identify intended beneficiaries, analyze the operating context of the initiative and understand the needs of stakeholders

Recommendation 1.1.1: Be aware of who are the initiative’s intended beneficiaries and have clear strategies to facilitate their active participation

Recommendation 1.1.2: Carry out a context analysis to understand the operating context of the initiative and identify the needs of its stakeholders, with particular regard to the intended beneficiaries

The appliers identify the intended beneficiaries and set clear strategies to engage them and enable their participation (in this regard, please refer to sub-criterion 2.1 and 3.2). In the MULTI-ACT Framework, society and patients are often the key beneficiaries. The appliers should explicitly identify these beneficiaries, their characteristics, and their needs. This step is necessary for the identification of the initiative’s long-term goals later on.

The initiative also conducts a context analysis. Its purpose is to identify the main actors and trends that may be challenging for the initiative, as well as risks and assumptions that may affect its performance. Context analysis involves looking at the current state of the “issue” that your initiative seeks to influence or the problem it seeks to solve: its social, environmental, and political conditions, actors who may be able to bring change. This is why, before defining the mission and agenda (see sub-criterion 1.2), appliers first analyze which “ecosystems” and communities that are affected, key issues and pressures faced, and the main social, political, economic ,and technological factors that together create the context.

It is recommended to carry out the context analysis in parallel with the Plan phase of sub-criterion 3.1, which describes profiling and mapping of the stakeholders.

Having identified the intended beneficiaries, analysed its operating context, and mapped its stakeholders, your initiative is ready to deepen its understanding of the stakeholders’ needs. Needs assessment is a fundamental process that leads to a better understanding of the challenges faced by the initiative and its stakeholders. You can use it to identify the change that your initiative wants to bring about in society. This change will be subsequently expressed through the initiative’s mission and detailed through its agenda, as described in sub-criterion 1.2.

The need assessment is also related to sub-criterion 2.2, which recommends initiatives to set up an initial consultation process to understand the bottom-up needs and challenges of the potential participants of the initiative.

It is possible to integrate the context analysis and needs assessment: appliers can identify the problem faced, its main roots, and its most relevant consequences, involving relevant stakeholders in this analysis. In the process, the stakeholders present their needs.

This exercise may facilitate the steps described in the following sub-criterion 1.2, namely the definition of the mission and the agenda.

Sub-criterion 1.2: Define a shared mission and common agenda

Recommendation 1.2.1: Define a shared mission and a common agenda involving relevant stakeholders, thus tackling the intended issue with a unifying long-term vision and a clearly defined set of objectives and actions necessary to pursue the mission.

Recommendation 1.2.2: Identify appropriate indicators in alignment with the initiative relevant aspects and objectives considering the different perspectives of the stakeholders involved.

Initiatives adopting MULTI-ACT Framework have in common the vision of striving to conduct mission-oriented research. They define their mission and agenda according to their specific vision and unique circumstances.

Mission definition

A mission statement defines your initiative’s current and future role, its goals[11], and its approach to reaching them.The mission statement includes:

•       Descriptive elements clearly illustrating what the initiative wants to achieve;

•       Transformative elements i.e. the changes the initiative wants to create in the context in which it operates.

With regard to the descriptive elements, your initiative may want to describe:

•       Its potential beneficiaries;

•       The scope of the intervention (e.g. health domain, geographical area, gender, socio-economic conditions).

With regard to the transformative elements, your initiative may need to clarify:

•       The expected change intended to happen for the beneficiary;

•       A baseline against which this change could be assessed.

Example of a research initiative mission[12]

Decreasing the burden of dementia by 2030 reducing the progression of the disease in affected patients in Europe. Descriptive elements: 1.      Beneficiary: affected patients 2.      Scope of the intervention: dementia brain disease in Europe Transformative elements: 3.      Expected change for the beneficiary: reducing the progression of the disease 4.      Baseline: the current burden of dementia.

 

Materiality analysis and identification of aspects

 

Figure 10: The materiality analysis as the bridge between the initiative’s mission and its outcome.

 

According to the criterion 5, appliers need to establish a shared and effective assessment system, and a mechanism to receive feedback. The assessment system must include a set of indicators and promote continuous improvement, and communication.

Then, the appliers enable the stakeholders to co-select measurable objectives in order to assess the progress and outcomes of the initiative. The initiative’s governance bodies, on other hand, identify the aspects of measurement through a process that requires identification of measurable and achievable targets: in this way, they ensure stakeholders’ engagement over time.

In order to assure coherence between the performance indicators used in monitoring and reporting and the interests of different stakeholder categories involved, the initiative carries out a materiality analysis.

Materiality analysis is a managerial tool that can facilitate the adoption of co-accountability and multidimensional impact assessment (MULTI-ACT Master Scorecard). It allows you to gather stakeholders’ perspectives and to identify the CRIF aspects that are significant for stakeholders. From this point of view, materiality analysis can be defined as a bridge between your initiative’s mission and the outcomes of the research it conducts. It links the reasons why the initiative was established with the results that matter most to the stakeholders. You will find detailed instructions on how to conduct it in the Materiality Analysis Chapter.

1)          Promoters [NM5] will be able to carry out materiality analysis via the MULTI-ACT Toolbox. Regardless of how the materiality analysis is carried out (whether it is via questionnaire, online survey, interviews, etc.), there are some general recommendations to be followed to get a robust and reliable analysis:

1)          Cluster the responses into different stakeholder categories. Results can be then aggregated following the suggested methodology hereby.

1)          Ensure the anonymity of responses in order to guarantee freedom of choice for the respondent.

1)          Define a minimum number of individual views required to be considered representative of a stakeholder category (e.g. min 5), in order to ensure balanced and veridical representation.

1)     Try to reach a heterogeneous cluster of responses within the same category: introducing additional specificities inside each stakeholder category enables to catch potential differences among the same clusters (i.e. in the case of a patient different level of levels of psychological attitudes toward the engagement).

1)          Provide complete guidelines and/or tools to a respondent that may not be fully aware of the initiative.

1)          Clarify a significance threshold under which the responses can be considered under-represented and thus not material.

Once the materiality analysis is completed, relevant aspects should be defined. In the example below (table 3), the materiality analysis identified 12 relevant aspects. As can be seen from the example, the number of relevant aspects can be lower than 15. Additionally, the promoter using the Digital Toolkit can decide whether to expand or reduce the number of relevant aspects.

Table 3 Example of a list of material aspects

MISSION	EXCELLENCE	ECONOMIC	SOCIAL	PATIENT REPORTED
Patient quality of life	Informing healthcare practice decision making	Improvement of health services	Political externalities	Patient Satisfaction
Drug supply to patient	-	-	-	Anxiety & Depression
Health services and products accessibility	-	-	-	Locomotion
Stakeholder engagement	-	-	-	Fatigue
-	-	-	-	Bladder function

 

Agenda definition

An agenda is a list of fundamental transformative objectives (i.e. priorities), including a description of the main outputs[13] and activities needed to achieve them. It is agreed upon by stakeholders and your initiative will aim to achieve its agenda in order to fulfil its mission. The agenda must be consistent with the aspects selected as relevant during the materiality analysis. For each priority of the agenda, the initiative formulates:

•       A transformative objective, describing the type of intervention and the transformative threshold and baseline according to which the initiative considers its intervention successful

•       Outputs and related activities needed to reach the transformative objective

Once they are defined, your initiative ensures proper dissemination and circulation among all involved stakeholders of the agenda, timeline, and objective, that should be shared among all team members.

Example of a research initiative[14]

Agenda of Dementia Care Initiative (timeline 2020 – 2030) •       Increase the percentage of dementia patients who are given personalized treatments, through the development of a customizable therapy protocol, according to specific patients’ needs, to be shared with an “X” number of medical facilities. •       Increase the dementia patients’ feelings of being more physically and intellectually independent through the development of a customised, free smartphone and computer IT application to be easily accessed by patients in Europe to perform daily tasks. •       Increase the percentage of early-diagnosed (within one year from the disease start) dementia patients in Europe through development of a digital application (e.g. background app linked to smartphone and computer) that is able to detect early symptoms of neurodegenerative diseases and recommend prompt treatment to users, to be available on at least on 2 operative systems (e.g. Android and IOS). The transformative objective (priority): The number of dementia patients who are given personalized treatments in Europe is increased by * %. Outputs: Development and adoption of a customizable therapy protocol according to specific patients’ needs. Activities: Research and development of the customizable therapy protocol. Assumption considered: •       If patients have personalized treatment, the progression of the disease could be slowed down up to * %. •       If patients are more independent in performing daily tasks, the feeling of the disease burden is decreased. Furthermore, performing these tasks could also be a stimulating activity to slow down the progression of the disease. •       If dementia patients are diagnosed earlier, the burden of the disease is drastically decreased due to the specific therapies that can be adopted.

When defining the agenda, always keep in mind the relevant aspects in order to ensure the alignment between the assessment system and the mission, agenda and objectives of the initiative. In the case of initiatives at an advanced stage of development, which have already defined and tested a mature governance model, an additional internal control system can be introduced in order to measure progress towards its agenda and the achievement of its transformative objectives. In this regard, the box below gives some further suggestions[15].

Timeline and Coherence Check

Once defined the agenda, the initiative should monitor the timeline of the intervention, namely the temporal and operation feasibility needed to achieve the objective. For instance, your initiative could answer the following questions: •       In what timeframe is it reasonable to reach our objective? •       Is it in line with our initiative’s timeframe? •       Is the threshold identified as our expected results realistically achievable? Can we contextualize the number? Did we make explicit the reference I am using to set up my percentage for my objective. Having defined the priorities of the agenda, the initiative should ensure the coherence and the causal link among activities, outputs, objectives, agenda and mission. In this regard the activity should lead to the output, the output - completely under the responsibility of the project - should lead to the objective. For instance, the initiative could answer the following questions: •       Is the agenda contributing to the mission statement? In which way? •       Are we accountable 100% over the activity and outputs? •          Are the activities contributing to the agenda? In which way? •

Finally, ensure secure funding to guarantee the maintaining of adequate resources to the development and the correct deployment of activities, as defined in sub-criterion 4.4. In particular, implement an effective cost‑management process: focus on the determination of the needed budget, cost analysis, and identification of gaps and critical issues.

Table Mission and agenda practical questions offers a set of additional data to be considered when defining the mission and agenda. In this last regard, please consider that the expected impact could be influenced by several factors both in and out of control of your initiative.

 

MULTI-ACT definition	Description	Question to answer	Timing	Sphere of control / influence
mission	The initiative’s current and future role, its goals and its approach to reach them	What is the long-term goal of the initiative?	Long term	Influence
agenda	The transformative objective, describing the type of intervention and the transformative threshold and baseline (according to which the initiative considers its intervention successful)	Why/What do I want to achieve? Which change do I want to contribute to/to bring about?	Medium- to long term	Influence
	The outputs needed to reach the transformative objective	How do I want to achieve it? Which concrete actions do I need to put in place?	Short term	Control
	Activity	Which activities will I perform?	Short term	Control

Table Mission and agenda practical questions

IDENTIFICATION OF INDICATORS[NM6] 

The materiality analysis leads to the identification of the indicators that the initiative should report in order to disclose its performances regarding the matters that most affect its stakeholders. The Master Scorecard links every aspect to a number of indicators to be taken into account. For instance, the initiative in the example identified Patient quality of life as a relevant aspect, it should report on:

 

The materiality analysis shows a static representation of the stakeholders’ priorities that might change over time. For this reason, the analysis should be carried out periodically, on a yearly or bi-yearly basis in order to ensure the alignment with stakeholders’ priorities and their commitment in the accomplishment of the initiative’s mission and agenda.

Sub-criterion 1.3: Promote a movement building approach to achieve transformative changes

Recommendation 1.3.1: Promote a movement building approach throughout all the initiative phases by enabling the generation of a community aspiration, becoming a platform that fosters change and innovation, engaging stakeholders in long term strategic action, enacting constant learning mechanisms and enabling authentic involvement of community

Recommendation 1.3.2 Be transformative and disruptive by promoting innovative problem-solving and critical thinking approach among involved stakeholders, in order to open new horizons for the research and go beyond the boundaries of the current research system, with the aim of achieving collective social impact

Appliers of the MULTI-ACT Framework should embody a movement-building approach[16] by integrating the above recommendations. In order to promote a movement-building approach and achieve transformative changes, your initiative:

·        Creates a sense of aspiration shared by the stakeholders in which everyone agrees and works together toward the achievement of the related goals;

·        Tries creating a “container for change” that seeks the change of the people involved in your initiative;

·        Engages in long-term (strategic) actions, at all stages of the project;

·        Focuses efforts on activities that result in a greater opportunity for change. This is achieved by having the agents participate and collaborate in long-term or strategic actions;

·        Incorporates a shared measurement process as part of a complete sharing learning process in which participant members “hold each other accountable and learn from each other’s successes and failures”[17]. In this sense, the shared impact assessment serves as a resource to provide feedback to the system and serve as a constant learning mechanism;

·        Ensures authentic community engagement including those negatively affected by certain measures in the process of change.

Sub-criterion 1.4: Guarantee ethical acceptability and social justice

Recommendation 1.4.1: Consider societal relevance and ethical acceptability of the initiative while minimizing potential unintended negative consequences

Recommendation 1.4.2: Aim to extend the positive impact of research to as many people as possible and ensure social justice

Appliers consider how relevant their initiative’s objectives are for the society and how to maximize its positive impacts while minimizing its negative consequences and ensuring that the rules of social justice are reinforced.

This recommendation is of qualitative nature and should be considered as a guidance and a reference to be applied throughout the entire process of decision-making. The responsibility of ensuring the consideration of this recommendation throughout the entire process could be assigned to the Compliance Committee, a body described in detail within sub-criterion 2.4.

Criterion 2: Participatory Governance

Appliers should guarantee an inclusive and equitable governance model promoting the involvement of all interested parties through a co-designing approach. To this end, ensure that the initiative:

•       Allows the involvement of private intended beneficiaries;

•       Adopts a multi-stakeholder approach enabling co-creation;

•       Implements a participatory structure;

•       Guarantees equity and mechanisms to avoid self-interest.

Sub-criterion 2.1: Allow the involvement of intended beneficiaries

Recommendation 2.1.1: Involve intended beneficiaries in the agenda design, in the decision-making process and in the initiative development, implementation and assessment. For the purpose of MULTI-ACT, patients are usually the intended beneficiaries. With specific regard to patients, develop a roadmap to capture “experiential knowledge” of patients, to better understand how to draw on their experience and use the experience constructively for co‐creation purposes and to evaluate the impact of research on the outcomes that matter to patients.

MULTI-ACT proposes a set of guidelines to support the engagement of patients which aim at leveraging patients together with the other stakeholders’ experience and at raising their ability to co‐create and participate to decision‐making processes.

The involvement of patients - defined as the intended beneficiaries - is pivotal in the implementation of MULTI-ACT Framework. In this regard, MULTI-ACT proposes a path for patient engagement to ensure that people affected by multiple sclerosis are given an equal voice with other stakeholders. To ensure continuous engagement of patients throughout the entire initiative and give them authentic influence, appliers:

1)     Appoint an Engagement Coordination Team (ECT), that will be in charge of coordinating the involvement of stakeholders, including patients, in all the operations. Initially, you (the Promotes) appoint the ECT, and the LB later accepts your choice or suggests a different composition. You will find more details on establishing, composition, and role of this crucial body in Governance bodies and in Establish the Engagement Coordination Team.

2)     Create a Patient Advisory Board (PAB), a specific group of patients within the Stakeholders Advisory Board (SAB), to be involved and engaged throughout the entire development of the initiative, providing advice, insight, and perspectives on the initiative’s activities and operations.

For the details about roles, responsibilities, appointment procedures, and structure of the above bodies, please refer to the Governance Bodies section. ECT is additionally discussed in a section of the Establish the Engagement Coordination Team.

The ECT [NM7] is part of the staff of the initiative and coordinates the participation of patients in the agenda design, in the decision-making process, in the initiative development, and eventually in the implementation and assessment phases. The ECT works as a facilitator and “floating” body between the Stakeholder Advisory Board and the Leadership Board. The ECT acts as guarantor and point of reference of patients’ participation in the initiative, it is in charge of the engagement processes and of all training and coaching activities that are preliminary to the patients and other stakeholders’ engagement. Furthermore, this team has not only the responsibility of patients’ engagement, but of all stakeholders whose participation is necessary to the initiative’s development.

The Initiative, according to its mission, might consider appointing a PAB. PAB is the body within the governance structure that presents the voice and opinions of patients, including underrepresented patients. It should be consulted and involved by the Leadership Board and the ECT during the key phases of the development of the initiative and when changes of any kind need to be implemented. It is part of a larger body, the Stakeholder Advisory Board.     

The promoters of the initiative should appoint the ECT in the setting-up phase of the adoption of MULTI-ACT Model with the agreement of the Leadership Board. During the constitution of the Stakeholder Advisory Board, carried out under recommendation 2.3.1, promoters should create the PAB, a specific sub-group, formed by patients. This structure is deemed to facilitate the involvement of patients and to give relevance to their stances and contribution to the development of the initiative.

Sub-criterion 2.2: Adopt a multi-stakeholder approach enabling co-creation

Recommendation 2.2.1: Prepare the initiative to implement co-creation processes by framing/reframing the composition of the initiative according to the new multi-stakeholder nature

Recommendation 2.2.2: Set up an initial consultation process in order to understand the bottom-up needs and challenges of the potential participants of the initiative

Multi-stakeholder approach to governance is essential for co-creation to happen. Co-creation may be defined as co-operation and learning from one other to raise awareness on important issues and to build relationships between groups and individuals[18], with particular attention to those that normally do not interact. In order to adopt the multi-stakeholder approach, your initiative needs to build participatory governance structures and processes, which are designed to create shared ownership of among its stakeholders (i.e. you – the promoter, patients, care providers – medical professionals, the industry, research institutions etc.). To shape the governance structure of your initiative that would be compliant with the multi-stakeholder perspective, first you need to identify the structure and tools best suited to help your initiative achieve its objectives.

To achieve this goal, the initiative first analyzes its current composition and envisions a stakeholder structure that would be ideal for achieving its mission and agenda. This activity allows to map the potential gaps in terms of stakeholder composition and to ensure that the initiative involves participants from all the relevant stakeholder categories. Once the initiative defines its composition, it identifies and considers stakeholders’ needs, challenges, and barriers to guarantee genuine participation.

In order to accomplish this goal, conduct the analysis described below:

1)     Analyse the current structure of the initiative, its organizational model, and its current participant composition (if your initiative already exists). Envision, what would be ideal for achieving your mission and agenda.

2)     Identify the stakeholders’ categories that could be involved according to the context and the objectives pursued by the initiative and, therefore, that could be potential participants in the initiative;

3)     Identify the potential relevant gaps in terms of stakeholder composition and, if applicable, integrate the participation of those stakeholder categories that are missing according to the above-mentioned point 2; Ensure that your initiative involves participants from all the relevant stakeholder categories.

4)     Identify and consider stakeholders’ main needs, challenges, and barriers to guarantee their genuine and committed participation.

Conducting this analysis is your task as a promoter. It precedes structuring of the governance model of the initiative itself, the composition of its bodies, and the formalization of the structure, participants, and roles, which will be explained in the sub-criterion 2.3: Implement a participatory structure and in the Governance Bodies section.

You will integrate the results of this analysis with the activities described in the Sub-criterion 3.1: Define and approve a methodology to engage stakeholders.

Sub-criterion 2.3: Implement a participatory structure

Recommendation 2.3.1: Define a clear and agile backbone structure and define clear roles and responsibilities of all involved stakeholders, based on the mission and the agenda

The participatory structure is the system by which an organization makes and implements decisions in pursuit of its strategic objectives. Appliers will need to adapt the structure to the organizational model proposed below or, if they are newborn organizations, define their structures accordingly. The section Governance bodies describes in detail the main bodies of the MULTI-ACT Governance Model: their main functions within the structure, process of appointment, and stakeholder composition. It is crucial that you become familiar with the content of this chapter.

The roles of the other bodies are further described under specific sub-criteria and in the Governance bodies section. In particular, the Leadership Board is described in the recommendation 2.3.2, the Working Groups in recommendation 4.1.1, the Engagement Coordination Team in sub-criterion 2.1, and the Compliance Committee in sub-criterion 2.4. You, with assistance from the ECT, are responsible for arranging an open call to interested stakeholders for participation in the Stakeholder Advisory Board (SAB). Establish rules for selection, composition, and balance of the SAB with the contribution of the Compliance Committee and the ECT.

Recommendation 2.3.2: Identify a mix of committed and skilled individuals that will be a part of the Leadership Board and balance them in terms of gender, sector background, geographical background, language, political diversity, opinion and experience

Set up the Leadership Board, comprising of at least one representative from each category of stakeholder (categories of stakeholders are defined in the recommendation 2.2.2 and sub-criterion 3.3). The composition of the LB should be balanced in terms of gender, sector and geographical background, language, political diversity, perspectives, and experiences. The members of LB should be committed and skilled individuals, which should ensure constant participation in the initiative’s development. The members of the LB should have equal power, in order to guarantee equity among participant stakeholders. The composition of the LB and its members should undergo the approval of the SAB and the PAB.

Specific activities, roles, and responsibilities of the LB are described and formalized within a procedure as pointed out in the recommendation 2.3.3 and in the governance bodies section. The LB appoints a chair/coordinator who will become the internal and external point of reference to the initiative. It may also create an operational team, such as a sub-board (the executive team) and a secretary (supporting operations).

Recommendation 2.3.3: Formalize how the stakeholders involved in the governance will interact with each other and cooperate within the governance structure

As an initiative, adopt a formal procedure, which will be public and will transparently define:

•       which is the governance structure of your initiative,

•       how the  governance bodies are composed,

•       how members are appointed,

•       how decision making-processes are handled,

•       how stakeholders and the public might participate in the initiative and/or take part in its governance bodies or in other bodies.

An example of how the procedure could be structured is reported below:

•       Roles and responsibilities

•       Structure and membership of the governance bodies

•       Operations (i.e. regular operations and meetings)

•       Relations between the governance bodies

•       External relations and public involvement

The LB has the responsibility of developing this procedure, with the support and contribution of the CC and the ECT. The resulting document should be shared and approved by the SAB.

Sub-criterion 2.4: Guarantee equity and mechanisms to avoid self-interest

Recommendation 2.4.1: Guarantee the support to and the meaningful participation of disadvantaged stakeholders (for financial, communication, language, cultural, age or mobility reasons) through appropriate mechanisms to give voice to each of them and avoid marginalization

Recommendation 2.4.2: Ensure that monitoring measures are put in place to protect the integrity and multi-stakeholder nature of the initiative and manage potential conflicts, considering that different views have to be accommodated in the decision-making process

Recommendation 2.4.3: Implement appropriate engagement mechanisms to create and maintain commitment and ownership among the participating stakeholders

To guarantee equity and implement mechanisms preventing self-interested actions of stakeholders, a specialized body is needed. To this end, you appoint the Compliance Committee (CC); this decision needs to be later confirmed by the LB and the SAB. It will be in charge of maintaining a balance among stakeholders’ influences and expectations and overseeing the ethical issues that may arise during the implementation of the initiative. More on the composition, appointment, and functions of the CC in the Governance bodies section.

The CC represents the point of reference for the implementation of recommendations 2.4.1, 2.4.2, 2.4.3, and with regard to those included in sub-criterion 1.4 and sub-criterion 3.4.

Criterion 3: Clear, effective and inclusive methodology of stakeholder engagement

Appliers of the CRIF are able to guarantee a comprehensive, balanced, and efficient stakeholder engagement process, ensuring the participation of patients and caregivers, and of other relevant stakeholders, by:

•       Defining and approve a stakeholder engagement methodology;

•       Engaging private intended beneficiaries;

•       Differentiating the level of engagement according to participants;

•       Ensuring a balance between the engagement of participants and agile management of the initiative.

Since MULTI-ACT is a collaborative tool, which requires the involvement of stakeholders in the entire governance process, this criterion works as an overarching principle for the other four governance criteria. In addition, the MULTI-ACT Patient Engagement Guidelines provide a methodology to engage the stakeholder “patient” and facilitate development of a roadmap to capture patients’ voices and help them to co-create with the other stakeholders’ experience.

Sub-criterion 3.1: Define and approve a methodology to engage stakeholders

Recommendation 3.1.1: Define a methodology to engage stakeholders, create and maintain an open dialogue with them and manage the engagement processes of participants throughout the entire design and implementation of the health research initiative

Recommendation 3.1.2: Provide clear information regarding why the initiative is engaging (the purpose), what issues to engage on (the scope), and who needs to be involved in the engagement

This fundamental process relates to the engagement of stakeholders who cooperate towards the achievement of the objectives of the initiative. The appliers define and implement a structured and detailed methodology to effectively engage those stakeholders who are of strategic importance, so they can cooperate towards the achievement of the objectives of the initiative.

Successful engagement depends on deep understanding why an organization is engaging (the purpose), what issues to engage on (the scope), and who needs to be involved in the engagement (the stakeholders). An engagement process should clearly describe:

•       How to establish commitment;

•       How to determine the purpose, scope, and stakeholders of the engagement;

•       How to integrate stakeholder engagement within the governance;

•       How to carry out the processes that will deliver quality and inclusive engagement practices, and valuable outcomes.

The methodology of engagement stakeholders should comprise at least some key phases, which can be summarized as follows:

1)     Plan – identify which stakeholders should be engaged in your initiative due to their strategic importance to achieve your mission. Cluster them into different categories that reflect different levels of engagement. Determine the rights, duties, and responsibilities for each category of stakeholders.

2)     Prepare – when you identify the stakeholders and determined the levels of engagement, assess:

a.      the different characteristics and needs that these stakeholders may have

b.     barriers concerning their effective engagement

c.      risks related to the involvement of such a diverse group of actors.

3)     Implement – define activities that will allow the participation of stakeholders in your initiative through formalized procedures that define in detail the interaction and cooperation between the different actors.

4)     Review and improve – put in place mechanisms that would guarantee the monitoring and evaluation of the stakeholders’ engagement in order to improve it.

In the Plan phase, the promoters:

1.      Profile and map your stakeholders: To design the stakeholder engagement process, you need a clear understanding of who the relevant stakeholders are, and how and why they may want to engage with your initiative. Profiling and mapping shall be reviewed and revised throughout the process and for this reason, it should be formalized. It is recommended to carry out the stakeholder profiling analysis in parallel with the context analysis as described in sub-criterion 1.1.

2.      Determine their levels of engagement: Map and cluster stakeholders into different categories to determine which groups and individuals are most important to be engaged from the point of view of the engagement process’s purpose and scope (please refer also to sub-criterion 3.4). Define different levels of engagement, which determine the different rights, duties, and responsibilities of the interested stakeholders. Defined levels are also used to establish the composition of the SAB (please also refer to sub-criterion 2.3).

In the Prepare phase:

1.      Build capacity: Different actors have different levels of expertise, confidence, and experience. Some individuals and groups may find it difficult to take up your invitation to engage, or their circumstances may hinder them from fully contributing to the process. This may be due to language, literacy, disability, or cultural barriers, problems of geographical distance, or lack of time, or gaps in their knowledge about a specific issue. The LB, with the help of the ECT, should timely identify where engagement capacity needs to be built , in order to avoid exclusion of these stakeholders, or to prevent them from disengaging (please also refer to the sub-criterion 3.2).

2.      Identify and prepare for engagement risks: In order to formally identify, assess, and address engagement risks, promoters you need to perform a risk assessment. The potential stakeholder risks could be, for instance: unwillingness to engage, participation fatigue, creating expectations of change that the organization is unwilling or unable to fulfil, a conflict between participating stakeholders, etc.

In the Implement phase:

1.      Invite and properly brief stakeholders: The LB ensures that stakeholders are invited to participate in the engagement activities in advance and that communications are appropriate for each stakeholder category. In order to mitigate the risks identified in the previous phase, ECT develops and provides the participants with the briefing materials and coaching needed to ensure the success of the engagement (please also refer to the sub-criterion 3.2).

2.      Develop an Engagement and Action Plan: The LB, with input from the stakeholders and the support of the SAB, establishes procedural and behavioural rules for the participants, which may include for example: guaranteeing that the opportunities for providing inputs are evenly distributed among participants, allowing all participants to express their opinion, staying focused on the transformational change that your initiative aims to achieve. Roles and responsibilities for all the participants have to be clearly defined, to regulate their cooperation and allow them to hold each other accountable. Based on the defined mission, create a collective Action Plan (please refer to the sub-criterion 4.1), adopted after consultation with all the participants of your initiative, to guarantee that it corresponds with the expectations of all relevant stakeholders.

In the Review and Improve phase:

1.      Monitor and review the engagement: The LB, in cooperation with the SAB, systematically monitors and evaluates the overall quality of the stakeholder engagement, including the evaluation of (please also refer to the recommendation 5.1.7):

a.      Commitment and integration;

b.      Purpose, scope, and stakeholder participation;

c.      Process (planning, preparing, engaging, acting, reviewing, and improving);

d.      Outputs and outcomes;

e.      Reporting.

2.      Learn and improve: The LB, in cooperation with the SAB and with direct inputs from stakeholders, if needed, continuously assesses the value of the engagement and improves its stakeholder engagement activities for stakeholders’ engagement. These processes need to be formalized to strengthen and optimize future activities.

The stakeholder engagement process is meant to be customized by each initiative which adopts the MULTI-ACT Framework, so feel free to adapt and develop it so that it fits your initiative’s specific needs. However, the above-mentioned phases represent the minimum requirements that you have to take into account to implement an effective stakeholder engagement process.

The above-described phases are supposed to be carried out by the promoters and the LB supported by the ECT. This is due to the fact that the first phase (Plan) is expected to be carried out when your initiative is being set-up, while the other activities occur when the Governance Model is being implemented, once the LB has been identified.

However, the appointment of the LB itself is carried out through a multi-stakeholder methodology. For this reason, you should follow the recommendations included in this sub-criterion when setting up the LB.

The ECT should support you, as the promoter, first, and the LB, later, during the entire process that will culminate in the definition of the stakeholder engagement process. This body will also be directly in charge of the implementation of the engagement methodology throughout the development of the initiative.

Sub-criterion 3.2: Engage intended beneficiaries

Recommendation 3.2.1: Guarantee the availability of customized training for lay participants (patients), who might not be trained to participate in complex research initiatives

Stakeholders such as patients are often involved in a research project as data providers (clinical trials, drug development) or users testing innovative technologies (biotechnological R&I), rather than engaged in the governance of R&I with decision making role. Each initiative adopting the MULTI-ACT Framework needs to involve this category of stakeholders to understand their needs and expectations and translate them into practice throughout the entire R&I process. You provide the right tools to all the stakeholders involved, so they are able to equally participate in all the steps of the process.

MULTI-ACT will exploit a training module to support the use of the MULTI-ACT Governance model in the management of multi-stakeholder research initiatives, including Patient Engagement: the MULTI-ACT Academy on Multi-Stakeholder Research Initiatives and Patient Engagement management (MULTI-ACT Training module®).

To successfully engage private intended beneficiaries, several activities need to be performed. These should be coordinated by the ECT, the body that will manage the process of involving several categories of stakeholders including patients, identified in sub-criterion 2.1.1. The main activities are described below:

1)     Setting in place the engagement process and providing the participants with the necessary briefing materials. These materials should contain a clear explanation of the initiative’s expectations concerning stakeholders’ engagement and facilitate communication between experts and lay participants; the materials should be made available in a timely manner. Make sure that aspects such as linguistic proficiency, disability, and literacy issues of stakeholders are addressed;

2)     Organizing training sessions in which private beneficiaries are transparently informed on the process and the role they play within it;

3)     Guaranteeing the involvement of private intended beneficiaries that may have experiences in multi-stakeholder initiatives to become the point of reference between the initiative and the stakeholder group.

The ECT focuses not only on the engagement of patients but trains all categories of stakeholders to ensure their fruitful engagement

Recommendation 3.2.2: Guarantee a fair and equitable process that takes into account the limitations that participants might encounter (e.g. cognitive impairment, behavioral issues, fatigue)

Science with patient input approach requires the active participation of patients in the governance, priority setting, and conducting of research, as well as in summarizing, distributing, sharing, and applying research results. A multi-stakeholder initiative can potentially engage a variety of stakeholders with different levels of expertise, confidence, and experiential knowledge. As explained in the recommendation 3.1.1, it is important to appreciate that some of the stakeholders may face obstacles to becoming engaged by your initiative or contributing to the process to the best of their abilities. Reasons range from lack of knowledge to life-limiting disabilities.

Another essential aspect to be considered is the fact that a research program/project within the health sector can be imagined as a path, namely a sequence of processes and activities in the R&I continuum where patients can be engaged in order to maximize the impact of R&I. R&I Path conceptualizes research as a sequence of processes and activities in the R&I continuum where patients can be engaged in order to maximize the impact of R&I. Consequently, after identifying the possible limitations that might be encountered in the engagement of patients, the appliers define if these limitations are the same for all patients involved across the R&I Path, or if there are some steps of the R&I Path which are more complicated and for this reason should be considered with more attention.

Following that, the actions to overcome these barriers and limitations need to be envisioned and, if not possible, alternative forms of engagement need to be discussed (i.e. engaging parents for children; relatives of people with cognitive impairments).

The ECT coordinates the participation of patients in the agenda design, in the decision-making process, in the initiative development, and finally in the implementation, monitoring and evaluation phases. Its facilitator role should guarantee that all possible limitations that might affect the effectiveness of patients’ engagement are taken into consideration and that mechanisms to avoid these situations are put in place. Indeed, it is extremely important that the R&I is carefully analyzed so that the ECT can be well informed and prepared on the possible limitations that this specific category of stakeholders might encounter in the several R&I steps, and carefully address them to guarantee an efficient and effective stakeholder engagement process. This activity also relates to the Prepare phase of the Stakeholder Engagement Methodology.

Sub-criterion 3.3: Differentiate the level of engagement according to involved stakeholders

Recommendation 3.3.1: Differentiate the level of engagement of involved stakeholders, considering:

•       their skills, capabilities and characteristics;

•       the stages and processes of the initiative;

•       the relationship with the involved stakeholders and their strategic importance to the initiative;

•       the resources available and the organizational constraints

Stakeholders engaged in multi-stakeholder health research initiatives have different skills, expertise, and interests. Once you mapped which stakeholders should take part in the initiative (refer to the sub-criterion 3.1), cluster them into different categories. Engage stakeholders according to their identified Levels of Engagement. In determining the Levels of Engagement, define the nature of the relationship you will develop with their stakeholders.

Cluster your initiative’s stakeholders according to their strategic importance, which could be based on the skills and the resources they might have at their disposal to achieve the mission of the initiative and to be accountable.

Stakeholders’ strategic importance for your initiative would then determine the Level of Engagement to be selected to best meet the needs, capacity, and expectations of the relevant stakeholders. Revise the Level of Engagement periodically, as they may change it over time as relationships deepen and mature.

• An example of levels of engagement is the following: [NM8] 

·          Co-design: stakeholders are engaged since the very beginning of the steps of the R&I Path with a decision-making role (i.e. they are part of the LB);

·        Involve: stakeholders are engaged in research project activities with an active role (i.e. they could be part of the SAB with specific roles and/or working groups according to their specific relevance);

·          Consult: stakeholders can provide feedbacks to decision-makers on their analysis and/or decisions, and they participate by being asked for advice and opinion (i.e. they could be part of the SAB and/or specific Committees);

·        Inform: stakeholders are informed about research priorities, activities, outcomes and impact of the initiative.

This clustering and prioritization effort will facilitate processes such as the election of representatives of each stakeholder’s category to be part of the LB, advisory bodies, or Working Groups. It will also be useful during the materiality analysis, when you engage the stakeholders based on the category and strategic importance, among others.

Sub-criterion 3.4: Ensure a balance between engagement of involved stakeholders and agile management of the initiative

Recommendation 3.4.1: Ensure that there is a right balance between an agile management process and the opportunities for engaging a wide range of participants. In particular, set in place processes to mitigate the challenges faced by collaborative groups, such as competition, conflict, cultural and behavioral differences, equity, resource sharing, communication, confidentiality concerns, and geographical dispersion

Identification of appropriate stakeholders to be involved in your initiative is essential to guarantee that there is a balance of different characteristics and backgrounds among participants, which is needed to achieve the transformational change. Moreover, it is fundamental that an initiative prepares appropriate mechanisms to deal with possible challenges that might arise due to the diverse background and characteristics of the stakeholders involved.

To mitigate the challenges that may be encountered by a collaborating group, the LB, with the support of ECT and CC:

•       Achieves a balance of interests in the subject matter and in the geographic scope among the participants within the governance bodies;

•       Strives for consensus on decisions that might define the milestones for the initiative;

•       Defines criteria in advance to determine when alternative decision-making procedures should come into effect, in case if consensus cannot be achieved. Criteria for determining when to consider voting could include those decision-makers who are not in the agreement. Initiative may want to provide alternative solutions and, if these are not accepted by the majority and a compromise is not reached, then alternative decision-making procedures could be implemented;

•       Defines a decision-making threshold (in relation to the voting process) to ensure that no stakeholder group or type can control the decision-making process.

The ECT guarantees and facilitates the participation of stakeholders with obstacles to engagement, encouraging and maintaining commitment, and ensuring a balance among different points of view. On the other hand, the LB should support the implementation of an agile management process.

These two principles might sometimes be in contrast: in this case, the cooperation between the ECT and the LB, with the support of the CC is fundamental to ensure a balance between the engagement of participants and the adaptive management of the initiative.

Criterion 4: Effective and efficient management and coordination of the initiative

Guarantee an effective, cooperative, and efficient coordination of the objectives and actions required to pursue the mission and the agenda. To achieve this goal, the initiative:

•       Enables cooperation and competition among participants;

•       Sets clear and transparent processes and timeline;

•       Maintains flexibility;

•       Ensures the presence of secure funding, solid organizational structure, and resource management.

Sub-criterion 4.1: Enable involved stakeholders to coordinate their efforts and perform activities

Recommendation 4.1.1: Put in place processes that allow involved stakeholders to perform mutually reinforcing activities and coordinate collective efforts to maximize results and create opportunities for change

One of the objectives of your initiative is to create accessible and innovative mechanisms to facilitate interaction and bridge the gap between stakeholders to collaborate. Consequently, the appliers also put in place processes that allow participants to perform mutually reinforcing activities and hold each other accountable through a clear definition of roles and responsibilities.

To allow participants to carry out mutually reinforcing activities, the LB should implement the following activities:

•       Definition of a collective Action Plan in line with the mission and agenda and specifies the strategies and actions that the different partners commit to implement to achieve such change;

•       Implementation of these strategies by all the participants to advance the shared Action Plan;

•       Establishment of WGs and other collaborative structures with the role to coordinate activities aligned with the Action Plan;

•       Setting up accountability mechanisms to hold partners accountable for implementing activities as planned;

•       Organization of Touchpoint Meetings to create opportunities for change, such as:

o   holding periodic events in order to discuss potential challenges, foster innovative thinking, and identify practical solutions;

o   hosting webinars to support stakeholders in the implementation of actions.

The LB is in charge of the implementation of the above actions. It defines the collective Action Plan and oversees that the defined actions are implemented by all the participants. The WGs are composed and balanced according to the stakeholders’ categories and the needs of your initiative. They are in charge of specific tasks (e.g. research or reporting activities, as described in criterion 5). WGs report to the LB. Both cooperation and competition within these bodies should be promoted: participants with different backgrounds, experiences, and interests should be involved in the implementation of a given task/activity to provide multi-disciplinary inputs while pursuing a common goal. This could provide an added value to the initiative itself since multi-stakeholder interactions are considered at all steps of the Research & Innovation Path.

Sub-criterion 4.2: Set clear and transparent processes and timeline

Recommendation 4.2.1: Identify and negotiate with stakeholders a consistent program/project timeline and schedule, in order to assure that the progress is soundly implemented

Recommendation 4.2.2: Commit to transparent, evidence-based decision making, in order to reach the objectives established in the mission and agenda

Recommendation 4.2.3: Guarantee a mechanism of review and evaluation, which allows to learn and improve the collaboration among stakeholders

The appliers define a timeline to assure that progress is soundly implemented and the organizational process is transparent. Moreover, they should define clear roles and responsibilities among participants to guarantee that each actor clearly knows their role, exercises their rights, and fulfills their duties. To implement an effective process, the collective Action Plan should also contain:

•       Clear and measurable targets to be achieved by the initiative;

•       A clear program/project timeline with achievable deadlines to allow participants to hold each other accountable and evaluate the progress achieved by the initiative over time;

•       A clear review process which will have to be carried out on a periodical basis to keep track of the achieved targets.

The definition of these rules and deadlines should be discussed and defined by the LB, because their implementation will be pivotal to guide the initiative in the achievement of its defined mission and agenda.

The implementation of the above activities is strictly related to the previous sub-criterion because WGs are the bodies responsible for carrying out the activities through which the targets can be measured and achieved. To facilitate this process, the LB can appoint a Secretariat or Management Team (please consult Governance bodies section) which will help to enforce deadlines, supervise activities, and improve your initiative’s performance as defined by mission and agenda.

Sub-criterion 4.3: Maintain flexibility

Recommendation 4.3.1: Maintain flexibility, adjusting the goals and implementation actions to the changing reality and needs

It is appliers’ task to stay up-to-date on the current needs of the beneficiaries. In the implementation phase of the research initiative, they should consider adjusting the goals of the initiative and which stakeholders it engages due to changing needs and reality. When adopting this recommendation, the initiative needs to adapt it to its specific needs and context. Several practices could be evaluated by the LB of your initiative to respond to current needs, such as:

•       Prepare a Progress Report (for example on a yearly basis) as it is a useful tool to collect all the achievements but also the concerns raised throughout the process by stakeholders and possible recommendations for the future (sub-criterion 5.1.6);

•       Organize a consultation event on a periodical basis where stakeholders can express their views and confirm their alignment with the defined agenda (sub-criterion 5.2.1);

•       Consider the review by external actors to identify possible gaps and areas for improvements;

•       Periodically review the mission and agenda according to the above-mentioned activities (sub-criterion 1.2).

These activities could be carried out by specific WGs or other bodies working under the supervision of the governance bodies. We recommend that they adopt flexible risk management. The structure of the initiative and the organization of the activities should be flexible enough to:

•       Allow for managing major changes that may arise within and outside the project;

•       Guarantee that the initiative is able to pursue the same transformative objective through a different strategy.

It is better to structure the initiative focusing on the objectives, rather than the activities, that may be reviewed following a potential external or internal change and according to the changing scenarios.

Sub-criterion 4.4: Ensure the presence of secure funding, solid organizational structure and resources management

Recommendation 4.4.1: Provide and maintain adequate resources (including financing, staff and technical expertise, and in-kind contribution)

Recommendation 4.4.2: Ensure that the internal team has solid skills to carry out the activities and cooperate with involved stakeholders

Recommendation 4.4.3: Adopt a cost management process and an efficient management to avoid inefficiencies

Recommendation 4.4.4: Maintain accountability over time keeping track of expenses and revenues

For an organization to accomplish its mission and carry out its operations, it is necessary to ensure that it is financially secure. To do so, it has to secure funding, a solid organizational structure with technical expertise, and solid resources management. To implement an effective cost management process, the LB may:

1.      Determine a budget: establish the amount of funding that your initiative has at its disposal;

2.      Conduct a cost analysis of the project: based on the timeline included in the collective Action Plan, understand the real costs that will be sustained by your initiative throughout the timeline of the project (including research funding, staff, and technical expertise, organization of meetings, other general expenditure);

3.      Identify possible gaps and critical issues in financial and resource management: identify potential critical issues and develop possible adjustments that would guarantee efficient management of the budget. Identify possible gaps and critical issues based on cost analysis. The Analysis should also propose some possible refinements that would guarantee efficient management of the budgeting to avoid inefficiencies.

The LB may choose to appoint a Secretariat or Management Team (sub-criterion 2.2) which will ensure financial security. Depending on the size of your initiative, it could also be supported by other bodies such as the CC and/or others. This process is conducted to ensure that your initiative is financially secure, running public accounting for expenditures and income, and ensuring that it operates in a legally compliant manner in relevant jurisdictions.

Criterion 5: Co-accountability assessment

Appliers establish a shared and effective measurement system, including a set of indicators, that promotes the improvement of operations and communication, and set a mechanism to receive feedback. This Criterion is connected to the Materiality Analysis and Master Scorecard, about which you can read in the respective chapters. To achieve this, the initiative will:

•       Develop a shared measurement and monitoring system;

•       Establish effective feedback mechanisms;

•       Guarantee continuous learning, communication, and disclosure of knowledge.

In this regard, a key step is materiality analysis (sub-criterion 1.2) that enable the initiative to align its activities in coherence with its mission and stakeholders’ perspective.

Sub-criterion 5.1: Define a shared assessment system

Recommendation 5.1.1: Enable the co-selection of relevant aspects, according to the different impact dimensions, in order to identify the topics that matter the most to the initiative and its stakeholders

Recommendation 5.1.2: Select appropriate indicators from the list of relevant aspects according to different impact dimensions and stakeholder perspectives in order to comprehensively assess the impact of health research

Recommendation 5.1.3: Ensure that the list of selected indicators consider the impact on patients

In order to define an assessment system that would be coherent with stakeholders’ perspective and would include the aspects the matter most to them, appliers consider the aspects chosen via the materiality analysis (recommendation 1.2.1).

In the customised Master Scorecard, the user is able to identify a list of indicators that allow reporting the initiative’s results in relation to different dimensions (efficacy, excellence, economic, social, and patient-reported dimensions). It is important to ensure that the list includes relevant indicators under the dimension Patients Reported Dimensions, indicators that are related to impact on patients directly reported by them without the intervention of the clinicians, such as the Patient Reported Outcome (PRO).

The LB is responsible for the definition of a shared assessment system, however it could nominate a committee to carry out the related activities.

Recommendation 5.1.4: Establish a shared assessment system consisting of a set of indicators consistently tracked over time and a shared data collection process

Recommendation 5.1.5: Ensure that the shared assessment system (Master Scorecard) is coherent to the mission and the agenda of the initiative over time, guaranteeing its alignment to stakeholder perspective

To establish a shared assessment system, the initiative defines a data collection process based on the indicators selected during the materiality analysis, which includes all the relevant stakeholders. The indicators need to be consistently tracked over time.

The LB ensures that there is constant alignment between the shared assessment system and the mission and agenda of the initiative. Periodically, when the agenda is updated, the shared assessment system needs to be updated as well.

Recommendation 5.1.6: Transparently report and communicate the initiative’s results and progresses to the public

Your initiative communicates its results and progress to the public in a transparent manner, through two complementary solutions:

•       A Progress Report published on a regular basis. The Progress Report is a document made available to the public that discloses information regarding the achievement (or non-achievement) of your initiative’s objectives and key performance indicators. In particular, the Progress Report discloses information regarding the indicators identified by the initiative in the sub-criterion 5.1.2, according to the aspects of measurement identified in the sub-criterion 1.2.2. The Progress Report contains general information regarding the management and implementation of the CRIF aspects and other relevant information regarding the achievement of the initiative’s mission and agenda. The Progress Report should be published on a regular basis, every one or two years, according to the specific circumstances of your initiative, and should be published online and made available to relevant stakeholders. An open platform, which includes a visualization of the performance of the initiative according to the identified indicators

•       An open platform is an online tool offering a visualization of the performance of the initiative according to the indicators identified by your initiative (in the sub-criterion 5.1.2, according to the aspects of measurement identified in the sub-criterion 1.2.2). The open platform offers access to key performance indicators regarding the initiative’s implementation. The platform contains general information regarding the management and implementation of the key aspects measured and other relevant information regarding the achievement of the initiative’s mission and agenda.

The LB is in charge of gathering information that will constitute the basis for the Progress Report, to create the open platform and to make these tools available to stakeholders and to the public. The LB may appoint a Working Group or a committee for this purpose, or use the help of the Secretariat or Management Team.

Recommendation 5.1.7: Constantly review the initiative according to the results of the assessment

Leveraging the performance assessment requires your initiative to establish a review process that will use its results. The assessment helps in improving performance and practices. To achieve this, the initiative will conduct a periodic review that includes at least the following activities:

•       Perform an analysis of the indicators on the initiative’s performance and results, emerging from the shared assessment process;

•       Set up an improvement plan identifying counteractions and improvement actions for the initiative;

•       If necessary, refine the agenda according to the results of the review.

A third-party actor could be involved in the process to ensure transparency and external oversight. The process should be open to the public to allow external stakeholders to provide suggestions and feedback. It should be implemented on a periodic basis (i.e. every 2 years), according to the needs and the characteristics of your initiative.

The review process should be led by the LB and the SAB, which might appoint a specific committee to carry out the operational activities linked to this process, or depending on the size of the initiative, the Secretariat or Management Team could be as well in charge of developing such activities.

Sub-criterion 5.2: Set effective feedback mechanism

Recommendation 5.2.1: Implement structures and processes allowing to inform, engage, and seek feedback from internal and external stakeholders, including concerns about the initiative and its development

The MULTI-ACT Framework attaches great importance to the initiative’s ability and willingness to receive constant feedback from internal and external stakeholders. Both are crucial to improving the efficacy and efficiency of the initiative, not to mention its responsiveness to the ever-evolving needs of stakeholders. For this reason, it is necessary that your initiative establishes a process that allows stakeholders to raise concerns and express their opinions. It is LB’s task to:

1)     Identify the most suitable and appropriate channels through which stakeholders can communicate and raise their concerns (e.g. email, website, letter);

2)     Set up the activities necessary to gather stakeholders’ feedback;

3)     Elaborate stakeholders’ feedback;

4)     Ensure that the feedback is appropriately managed and considered within the review process, under recommendation 5.1.7.

The initiative encourages stakeholders to provide feedback on the implementation of the initiative and keep them informed about the process in place to consider their concerns and integrate their feedback. Channels for feedback may be individual-based (e.g. anonymous hotline, web-format to be filled in) or participative (e.g. working groups, stakeholder consultation processes).

The initiative reports formally on how your initiative analyzes, manages, and integrates stakeholders’ feedback. The implementation of this sub-criterion should foster the review process carried out under recommendation 5.1.7.

The LB, supported by the Secretariat or Management Team, is in charge of setting up a process to collect concerns and opinions from stakeholders. The ECT participates in this process and is in charge of maintaining the active participation of internal stakeholders.

Sub-criterion 5.3: Ensure continuous learning, communication and disclosure of knowledge

Recommendation 5.3.1: Establish processes for continuous learning to improve the research evaluation framework and engage the public and the community, building trust among all involved stakeholders through constant communication. Ensure the existence of mechanisms for transparency and prioritize clear, accessible internal and external communication

The appliers need to build a trustful and continuous relationship with the public and the communities with which your initiative interacts. This can be achieved through a constant, clear, and useful flow of information. To implement this recommendation, the LB ensures that:

•       Communication on the most salient activities of the project is made public;

•       Communication is clear, accessible, and useful. It is made available to stakeholders according to their specific needs.

The initiative can use “unilateral” tools, such as newsletter, website, blogs, reports, but also “interactive” tools, such as training courses, thematic events, peer learning processes, practical guides for users, in-person meetings, events, and others.

The LB is to ensure a constant communication process with other health initiatives that may benefit from (or contribute to) your initiative itself. The LB is in charge of identifying opportunities for information exchange and cooperation and developing the most appropriate means to ensure these relationships in collaboration with the SAB.

The LB might appoint a specific committee to carry out these activities.

Patient Engagement Guidelines

MULTI-ACT Patient Engagement Guidelines aim to address the Criterion 3: Clear, effective and inclusive methodology of stakeholder engagement by providing a strategy to empower the stakeholders-patients to be engaged in research & innovation at the same level of the other stakeholders and to empower all the stakeholders to collaborate and co-create with the patients. The Patient Engagement Guidelines cover actions from establishment of governance bodies in charge of patient engagement, formulation of appropriate plans for patient engagement, a catalogue of methods for stakeholder engagement, and finally – monitoring and assessing the impact of patient engagement.

The figure below illustrates how patient engagement relates to transformational mission and governance bodies explained earlier. It is important to always ground the engagement process in the mission. Governance bodies are then responsible for conducting engagement at all stages of the R&I Path. The Patient Engagement Guidelines are ultimately about raising the return on engagement for your initiative.

Figure 11 Patient Engagement: from transformational mission to the raised value

 

How to use Patient Engagement Guidelines?

The Patient Engagement Guidelines provide you with a Patient Engagement Roadmap to capture, understand and draw on the experience of patients for the purposes shaping the research process together with them. You implement the Patient Engagement Guidelines by following this roadmap.

As the promoter, you are responsible for your initiative following the below steps, described in detail further in the chapter.

1.      Establish an Engagement Coordination Team (ECT) in charge of management of stakeholder engagement and organize the training modules for the ECT.

2.      The ECT defines the phases of R&I Path in which Patient Engagement is instrumental in achieving the mission and agenda of the initiative (see: Research & Innovation Path).

3.      Develop Patient Engagement Plans for the steps of the R&I Path identified in the previous step.

4.      Identify indicators to monitor and assess the Return on Patient Engagement, to verify if it has reached the expected impact on the initiatives.

 

Figure 12 Patient Engagement Roadmap

Establish the Engagement Coordination Team

Establishing the Engagement Coordination Team is a pre-requisite for effective use of these guidelines. As a Promoter:

·        Ensure that the governance structure, boards and processes of the initiative enable effective patient engagement. Ask yourself: Does the governance structure and process in charge of the Patient Engagement meet the MULTI-ACT Governance criteria?

·        Define the requirements for appointment of the ECT. It needs to be an open and inclusive process, emphasizing expertise needed from the members to do their jobs. Ask yourself: What is the role and expertise required for ECT? What training does the ECT needs?

 

Composition and skills of Engagement Coordination Team

The Engagement Coordination Team:

·        Designs Patient Engagement Plan, keeps it up-to-date and assures its financial sustainability, monitoring risks and proposing mitigation actions.

·        Designs and puts in place actions to engage patients in the initiative and constantly monitors and Patient Engagement performance and measures return on engagement;

·        Directs and moderates patient’s experiential knowledge so that it is transformed into outcomes that matter to patients and are “scientifically” validated as patient-reported outcomes.

·        Motivates patients to stay engaged along the initiative by assuring that patients “feel valued” by facilitating team interaction and setting up an inclusive research environment. ECT members need to be able to create commitment among the initiative members and their community.

·        Moderates the dialogue between interdisciplinary and different (and sometimes competing) voices and experiences and setting up a dispute resolution system;

·        Mitigates challenges such as ethical conflicts in protocol design, tokenism, power struggles, difficulties in recruiting different patients, additional time, cost;

·        Translates technical information into lay language that patients need in order to provide their feedback;

Training of Engagement Coordination Team members

The ECT is expected to be a unique board of experts with innovative functions, knowledge and expertise. If they are a new team, they will require innovative training. In order to allow the ECT to integrate patient experiential knowledge in research of your initiative, you should design and provide to the ECT a training module.

Make sure that the training module includes:

§  Adequate information about the project's mission and strategy;

§  Explanation of what is expected from patients and other stakeholders;

§  Explanation of what are the expected outcomes of the multi-stakeholder initiative;

§  Explanation of how these outcomes relate to the patients' needs in the given disease area;

§  Basic knowledge about innovative communication, learning and co-working techniques, and evidencing the value of patient and stakeholder engagement.

Information on methods for patients’ engagement should be integrated with examples of application in real cases for each method. The training should focus on the ability to elicit and capture patients’ stories and translate them into experiential knowledge. Plain language should be used and the content should be kept simple. Respect for human rights and dignity of the patient should always be considered.

Patient engagement in the Research & Innovation Path

Although patient involvement is crucial at every stage of the research, it is advisable to verify in which steps of the of the R&I Path it is best to engage patients to maximize the impact of your research. The steps of the R&I Path represent stages in research and management of funding and performing research within health multi-stakeholder research initiatives or RFPOs.

As described in the Plan phase in the sub-criterion 3.1 and in the recommendation 3.2.2, it is equally important that you identify obstacles that the patients may face in becoming fully engaged and contributing.

The steps of the R&I Path represent stages in research and management of funding and performing research within health multi-stakeholder research initiatives or RFPOs. They differ slightly for the Governance Program Level, which concerns often complex research programs, and for Project Development Level, which concerns single research projects.

Figure 13 Research and Innovation Path

Although patient involvement is considered crucial at every stage of the research, it is advisable to verify in which steps of the of the R&I Path it is best to engage patients to maximize the impact of your research. The steps of the R&I Path represent stages in research and management of funding and performing research within health multi-stakeholder research initiatives or RFPOs.

Governance Program
Breaking down the boundaries	Conditions that should be set in RFPOs in order to facilitate patient engagement as standard practice, e.g. patients help to review patient engagement policies and guidelines.
Setting research priorities	Actions to raise interest in a specific research domain, its importance, priority or rank. E.g., patients advance their interests in a specific research area.
Steering institutions	Actions performed to establish governance bodies. E.g. patients are invited to be member of committees and boards.
Design and planning	Design and planning of all the activities that lead to implementation of a concept or idea, and which help to achieve initiative’s designated objective. Patients are engaged in the development and monitoring of research programs
Executing research	Activities to perform the research program or a specific research project for the purpose of achieving the initiative’s designated objectives. Project Development (see below) level takes places at this stage.
Evaluating research	Activities to determine the value created by a research program or project, establishing their outputs and outcomes, the degree to which their pre-established goals were achieved, and their impact. Patients are engaged to working with other stakeholders on research reports
Translation to community	Activities to foster and facilitate the uptake of results of research programs or projects within wider society. Patients are engaged in the development of guidelines and advocacy activities

Table 10 R&I Path's steps (Governance Program)

Project Development
Design & plan	Design and planning of all the activities that lead to implementation of a concept or idea, and which help to achieve initiative’s designated objective. Patients are engaged in the development and monitoring of research programs
Conduct & operate	Conducting & monitoring project (e.g. ICT device development).
Evaluation	Activities to determine the value created by a research program or project, establishing their outputs and outcomes, the degree to which their pre-established goals were achieved, and their impact. Patients are engaged to working with other stakeholders on research reports
Translation to community	Activities to foster and facilitate the uptake of results of research programs or projects within wider society. Patients are engaged in the development of guidelines and advocacy activities

Table 11 R&I Path's steps (Project Development)

 

The list of actions associated with the R&I Path steps is given in the Menu of Patient Engagement Activities below.

7-steps R&I Path	Menu of Patient Engagement Activities
BREAKING DOWN BOUNDARIES	Patients help to define what are the boundary condition for patient engagement in your multi-stakeholders initiative Patients help to provide an overview on the facilities and infrastructure they need to be engaged in the R&I. Patients help to review patient engagement policies and guidelines
RESEARCH PRIORITIES	Patients are engaged to co-design research agenda Patients are engaged in advancing their interests in a specific research area Patients are engaged to prioritize research objectives
STEERING INSTITUTIONS	Patients are invited to be members of committees and boards; they provide guidance on key issues such as company’s policy and objectives, budgetary control, marketing strategy, resource allocation, and decisions involving large expenditures Patients are invited to advise the steering and advisory committees Patients are engaged in defining ethical issues, anticipating risks and barriers for patient engagement in governance bodies
DESIGN & PLAN	Patients are engaged to suggest endpoints and outcomes of research Patients are engaged to propose specific objectives of research Patients are engaged to define the relevance and acceptability of proposed research to patient community
RESEARCH EXECUTION	Patients are engaged in the development and monitoring of research projects (e.g. collaborating for ICT device development, for the enrolment to increase participation and decrease drop-out rate, to increase compliance with protocols and facilitate data collection, for writing and review of publications. Patients are engaged in development and monitoring of research programs (e.g. release of calls for proposals, selection of projects to be funded, monitoring of funded projects).
EVALUATION	Patients are engaged in discussions in multi-stakeholder teams about new methods to measure the impact of research Patients are engaged in assessment of new approach and products arising from research Patients are engaged to working with other stakeholders on research reports
TRANSLATION TO COMMUNITY	Patients are engaged in shaping the ‘translation strategy’ of research results into easy-to-use and easy-to-understand (lay) material and in communication activities to disseminate the research results Patients are engaged in the development of guidelines and advocacy activities Patients are engaged in advocacy to leverage uptake of the research results

Table 12 the Menu of Patient Engagement Activities along the Research and Innovation Path

Patient Engagement Plan

The Patient Engagement Plan is framework that allows your initiative to plan patient engagement in a systematic manner consistent with progress towards fulfilling the mission. The MULTI-ACT Governance Model offers a practical template to support design of the: you can find it in the Appendix 3: Patient Engagement Plan Template[NM9]  or use the Patient Engagement Plan Tool in the Toolbox. Its purpose is to provide you with a tool to integrate patients’ experiential knowledge into your R&I initiative, bringing expertise and knowledge complementary to the ones of other stakeholders. Patients, as members of the ECT and key stakeholders, develop the Patient Engagement Plan with the other stakeholders, ensuring representativeness of their community.

The assessment of the implementation of the Patient Engagement Plan is aligned with the Plan phase in the sub-criterion 3.1. The description on how to develop the Plan is presented below.

The indication included in the template found in the Appendix 3: Patient Engagement Plan Template is not necessarily the norm or a common practice, but rather a first attempt to provide practical guidance to RFPOs on how to plan, launch and monitor their Patient Engagement actions. Each mission is unique and requires ad-hoc interventions.

Minimum requirements of patient engagement plan

The Patient Engagement Plan aims to be a managerial tool and support operationalization of patient engagement in research. The plan should comply with the Patient Engagement criteria, which you can find in the Appendix2: Patient Engagement Criteria , and meet at the minimum following requirements:

§  Select actions of patient engagement that needs to be implemented in order to achieve the vision of the project;

§  Define roles and responsibilities of the team that should manage and carry out the implementation of such Patient Engagement actions;

§  Design methods to value and acknowledge patients’ experiential knowledge, including establishment of appropriate recognition of their contribution, and avoid tokenism;

§  Choose clear and measurable targets for performance and Return on Engagement;

§  Present clear timeline of activities and sustainable budget;

§  Define a clear review process (e.g. report on the performance and Return on Patient Engagement);

Process to design the Patient Engagement Plan

The design of the Patient Engagement Plan relies on the following steps:

1)   Define the purpose of engaging patient, keeping in mind your initiative’s mission.

·      Considering the given mission, how can patients help to meet the challenges (‘utility in context’)?

·      Which priorities of the mission/agenda would benefit more from patient engagement activities?

2)   Define expectations and objectives in relation to patient engagement in the R&I Path: Levels of Engagement, Type of patients, Requirements, Discussion questions.

·      In which of the R&I Path do we need to engage patients?

·      What do we expect from patients?

·      What are the specific Patient Engagement action plans for each of the steps of the R&I Path identified as relevant?

·      What type of patients do we need to engage?

·      What discussion questions can be used to capture patients’ experiential knowledge?

3)   Define potential risks and mitigation plans

4)   Define measurements to assess the performance and return on engagement.

5)   Define clear and effective training of patients and researchers, to prepare patients for their involvement and in particular for taking part in conversations with researchers that support mutual learning (in line with the role of the ECT).

6)   Design a system of recognition that acknowledges patients’ contribution to research, and the value of collaboration (e.g. expenses reimbursed, results co-authored by patients, patients in peer-reviewers and open access research).

7)   Define budget for putting into effect of the Patient Engagement Plan

8)   Define reporting, meeting and communication channels. The reporting should include mechanism to monitor and evaluate the performance and value of Patient Engagement ex‑post and during implementation (e.g. describe the review process in relation to the performance and value of the Patient Engagement, describe how the objectives for Patient Engagement are met, performance and Return on Patient Engagement). Define the value of Patient Engagement (Patient Engagement Plan/ Cost to put in place the Plan = Value). The reporting and monitoring system shall be defined on the basis of the shared vision (criterion 5).

9)   Check that the Patient Engagement Plan complies with ethical regulations

10)    Check the coherence of the Patient Engagement Plan with the MULTI-ACT Governance Model, criteria and minimum requirements.

11)    Define technicalities and operational aspects to enable a supportive research ecosystem.

 

Methods to engage patients

The following paragraphs contain recommendations on methods of patient engagement. Assessment of the implementation of the Patient Engagement Plan aligns with the Prepare phase of the Sub-criterion 3.1: Define and approve a methodology to engage stakeholders.

General recommendations for establishing the conditions for an effective partnership between the members of the Engagement Coordination Team

Creating right conditions

When selecting patients and other stakeholders as partners, take into account communication skills, motivation, and constructive assertiveness in a team setting.

Patients and other stakeholders have to be present from the beginning of the R&I process and considered on the same level as the other members and professionals of the Engagement Coordination Team, despite the differences in the contributions they can make means that they want to "learn" but also to "teach" others in a co-learning perspective.

It is important that ECT establishes a supportive research ecosystem leveraging Patient Engagement (communication channels, resources, infrastructures, organizational/institutional) to the ECT needs to assure that patients understand and agree on the research agendas, and to assure that they feel comfortable and recognize their unique perspective (motivation).

Strengthen the team spirit by creating a supportive environment that promotes partnership and open dialogue.

To allow full participation, the patients should be placed in the best possible conditions with regard to the needs. Respect and consideration must be guaranteed, so as to manage prejudice and hostile behaviour by the medical professionals towards the highly disabled patient population.

It is the initiative’s role to empower the patients to play their role in the team, first of all through comprehensible information, in their native language. There also should be adequate compensation in terms of reimbursement, giving the knowledge back and training and others. Their expectations of the engagement and what they hope to achieve must also be considered.

In the Engagement Coordination Team's work process, it is useful to increase public awareness by providing open access publications and sharing the results with the patients’ community. Prior to release of the research outcomes, a final discussion with patients will assure consensual decisions.

Seek partnerships with culturally specific groups to ensure compliance among different cultures and geographical settings, especially underserved populations. Cultural context, beliefs and traditions influence patients' lived experience and provide additional insights.

You can find more tips and principles to follow when engaging this special stakeholder category in the recommendations 3.1.1 and 3.2.2.

Using the right methods

In practical terms, the best way to engage patients is to use mixed methods: offline (face-to-face) methods (engagement without using computers, smartphones, tablets, or other internet-connected device/digital systems) and online methods (engagement through computers, smart phones, tablets, or other internet-connected device/digital systems).

Online methods make it possible to gather patients’ perspectives on a global scale while offline methods are useful to facilitate patients in providing their experiential knowledge as they may feel more comfortable to express their feelings face-to-face and they may be supported by a professional skilled managerial team (i.e. the ECT). Moreover, offline methods allow stakeholders to discuss more in-depth and to establish and to maintain a good partnership with patients. In particular, the ECT works mainly offline and they may use online methods to reach their community and a large consensus.

IT tools are useful for joining and recording conversations, proactively resolving complaints, promoting transparency, and enhancing patient experiences. It also requires organizations to comply with meaningful use criteria, such as engaging patients and families in their care, improving quality and care coordination, and reducing disparities[19]. Use of social media may affect patient engagement and satisfaction in healthcare and research. Integration of social media into clinical practice and research can empower surgeons to synthesize effectively a patient support community that augments patient engagement and satisfaction[20]. The same may apply as well to the R&I domain and environment.

Social media may play a role in identifying patient insights and engaging them in R&I for the purpose of capturing their experiential knowledge. Evidence related to the efficacy and effectiveness of social media in this function is currently limited. Various challenges related to privacy and security concerns, usability, the manipulation of identity, and misinformation have also been identified [21]. You have to exercise caution in their use and investigate, whether the way to envision to employ them for patient engagement has a scientific basis. Use of social media and social networks for science and research as a method to capture patients’ voice is worth investigating for a start[22], even though it has not been scientifically validated.

Review and ranking of the most appropriate methods for the engagement of patients and other stakeholders

Below you will find descriptions of the methods selected by Multi-Act as appropriate for the engagement of the public in decision-making processes in the R&I, and in particular of patients and other stakeholders. These methods are the following: Focus Group, Democs Cards Games, World Café, Consensus Conference, Community Advisory Board, Delphi Method, Citizens Hearing, Serious Gaming. Many of the methods have a versatility to be used both on- and offline.

The list is by no means exhaustive; your choice of the method depends on:

§     the goal of the specific engagement event – what you want to get out of it

§     the stakeholders being engaged: how much time and effort they can contribute, what obstacles they may face

§     what resources you have assigned for the occasion (monetary, human, time etc.)

§     how familiar you are with the technologies to be used

Focus Group and Democs card game are useful for capturing experiential knowledge and Give Voice to Patients . In line with MULTI-ACT Model, these two methods are considered "good methods" that the ECT could apply to engage patients and stakeholders in R&I.

Focus Group

The is a qualitative method which is used to determine the preferences of people or to evaluate strategies and concepts. The method has originally been designed for market research. Focus group is undoubtedly the most widespread technique of engagement. It is rooted in qualitative studies, where it is a standard way of gathering patients’ input and learning about their views and experiences. Its scope of application has widened in recent years, with the method being used for decision-making and guidelines formulation[23], not without some criticism regarding insufficient separation of these two functions. Participants are selected according to certain characteristics in common that relate to the research topic and are grouped into 8-10 people. It can be conducted face to face or in virtual digital space. The method is often used to generate or evaluate hypotheses and ideas in conjunction with a quantitative method, or as a primary data-collection method. Example: Selected patients and stakeholders are invited to a meeting to discuss about a topic.

Democs

It is both a card game and a policy-exploration tool that enables small groups of people to engage with complex public policy issues. It aims to help people find out about a topic, express their views, seek common ground with other participants, and state their preferred policy position. There are already a number of Democs kits on different issues which can be bought or downloaded for free from New Economics Foundation (NEF) and Play Decide. Example: Patients are provided with discussion cards that help them to express their views on a topic, to seek common ground with the other participants, and to express their preferences.

Table 13 Engagement methods: Focus Group and Democs

Below is a brief presentation [NM10] of the other suggested methods, which is based on the descriptions contained in the Engage2020 – Action Catalogue.

World Café	It is a method for engaging groups, both within organisations and in the public sphere. World Cafés are based on seven design principles and a simple method. World Cafés should offer an antidote to the fast-paced fragmentation and lack of connection in today's world. It is founded on the assumption that people have the capacity to work together, no matter who they are. Research indicated that World Café was not a popular method of engaging patients in the healthcare context, although some examples emerged. This may be in part due to the open-ended feature of the method. It is suitable for generating and sharing ideas, but does not guarantee a structured result, and does not support structured decision-making. [24] Example: A selected group of patients and stakeholders are invited to share their vision and position about a topic in a friendly space, and are encouraged to provide contribution to the debate.
Community Advisory Board	The Community Advisory Board (CAB) is a working group where patient advocates leaders from all world regions, work together to improve outcomes of patients covering patient information, research priorities, access to treatment and capacity building in the patients’ community.[25] The CAB method is used in leukaemia communities and by the HIV movement. Example: Patient advocate leaders are invited as member of the working Group to work on a topics.
Delphi	The Delphi method is a multiple iteration survey method that enables anonymous, systematic refinement of expert opinion with the aim of arriving at a combined or consensual position. Its purpose is to generate discussion and enable a judgement on a specified topic to be made so that policy decisions can be taken which can claim to represent a given group's wants and views. Along with modified Delphi Method, it emerged as the second most popular patient engagement technique after Focus Group. Initially designed for panels of experts to arrive at decisions without influencing one another, it is increasingly used for including patients, either forming their own panel, or together with experts and other stakeholders (e.g. community, healthcare professionals)[26]. Delphi can be applied online and it often is. Delphi Method appears to be a popular tool for prioritisation of core-outcomes in patient-centred guidelines [27], often in multi-stakeholder initiatives. Example: anonymous patients answer to multiple surveys to express their opinion about an approach defined by experts.
Consensus Conference	The purpose of this method is to enrich and expand a debate on a socially controversial topic. A group of citizens gather and set the agenda and the basis for assessment within a problem area. In the medical field, consensus conferences gathered practitioners and experts to build a consensus on either health knowledge (e.g. diagnostic criteria) or practices (e.g. best practices, treatment protocols). The format of these consensus conferences differs from event to event and cannot always be equated with the Consensus Conference engagement method, which has wider application. This literature review found papers describing engagement of patients using the consensus conference method in the course of research with the view of formulating guidelines or core outcomes. Example: A series of public events are organized to gather patients’ opinions about a topic and may result in a position paper.
Citizens Hearing	“The purpose of a citizens hearing is to inform and create discussion among citizens. The method uses brainstorming, dialogue, prioritization, reasoning and voting. Through dialogue and without interference of either experts or politicians, the citizens formulate their own suggestions and ideas (as to how a political (technological) problem can be dealt with) and present them to politicians”[28]. Some examples show how citizen hearing has been used to investigate the preferences of patients with respect to specific issues such as for example the use of health data and the status of health rights . This method showed enhanced understanding and awareness of the barriers to achieving positive solutions to help overcome them; and seek commitment on a joint plan for monitoring and acting on the topics. Example: Patients brainstorming, dialogue, reason and voting about a topic, without interference from any experts.
Serious Gaming	“The primary objective of ‘serious games’ or ‘applied games’ is to train and/or educate the user. These games serve as tools for acquiring complex knowledge in fields such as health care, education, engineering, city planning, emergency management, etc. Some serious games simulate real-life events and/or processes, thus providing the user with a problem-solving training environment. Furthermore, ‘serious games’ can be used in order to develop innovative products and services.”[29] Example: Patients are trained with an ICT game that presents the problem in a simple and fashionable way. The game is structured to provide patients with a training environment for problem-solving.
Research Studios Method	This method allows researchers to work closely with community members as they design studies. In 2009, the Meharry-Vanderbilt Community-Engaged Research Core began testing new approaches for community engagement[30], which led to the development of the Community Engagement Studio (CE Studio). This structured program facilitates project-specific input from community and patient stakeholders to enhance research design, implementation, and dissemination. Developers used a team approach to recruit and train stakeholders, prepare researchers to engage with stakeholders, and facilitate an in-person meeting with both. Literature reported that input from stakeholders was valuable and that the CE Studio helped determine project feasibility and enhanced research design and implementation [31].
Scenario Workshops	An instrument for participatory planning, it is based on dialogue and collaboration between local citizens, stakeholders, experts and policy makers. The method aims to stir dialogue, provide the opportunity for exchanging experience and knowledge, and facilitate consensus on proposed solutions among. It is a ”two-days meeting involving 25-30 local multi-stakeholder representatives to assess different solutions to a specific problem. Before the workshop, a set of scenarios is developed and used as visions and inspiration at the scenario workshop.”[32] Example: A Scenario Workshop is organized to discuss in a multi-stakeholder group on a specific R&I problem. The assessment of the different solutions proposed by patients and stakeholders results in defined and agreed actions to solve the problem. Patients comments on the scenario based on their experiential knowledge.
World Wide Views	The method is designed to closing the gap between citizens and policy makers in the context of global policy-making. Citizens at multiple sites debate the same questions on the same day. They are given materials before and during the day and then vote to choose pre-defined questions. “The votes are collected and reported online for comparison. It is possible to compare the votes across countries, continents, gender, age and other criteria. The results are analysed and presented to policy-makers.” [33] Example: A World Wide Views is organized to gather patients’ votes on a set of predefined research questions and results are used by policy-makers to design R&I and healthcare policies.
Voting Conference	Used in small settings and with diverse target groups, it is an approach similar to World Wide Views. E-conference (temporary online forum on a specific topic) can be used as tool. [34] Example: A Voting Conference is organized to collect patients’ votes on a set of predefined research questions and results are integrated in R&I activities.
Deliberative Polling®	Developed by James Fishkin, the public consultation method which combines deliberation in small group with scientific random sampling. It informs public policy. [35]
Deliberative online forum	Web-based (in online forums) discussions between informed individuals about issues which concern them, leading to some form of consensus and collective decision.[36]
Deliberative Mapping	Involving both specialists and members of the public, it combines varied approaches to assess how participants rate different policy options against a set of defined criteria. The method allows substantial involvement of public participants. [37].
Deliberative Workshops	Events with a focus on in-depth informed discussions on a complex or controversial issues to inform policy and regulation, exchange opinions or raise awareness. This method has also been used to develop research agendas and objectives.[38]. Example: Patients are engaged in deliberative surveys, small group discussions, online forums, dialogue events, etc. to express their opinions on specific R&I’s questions and issues and the results are used for deliberating on specific R&I policies. Patients can also rate different policy options against a set of defined criteria.

Table 14 Other suggested engagement methods

Measure the performance and effectiveness of patient engagement

To maximize the impact of patient engagement, the ECT identifies indicators suitable for performance measurement and assessment of the effectiveness of patient engagement in your initiative’s R&I processes. The assessment should combine quantitative and qualitative evaluation. The assessment of the implementation of the Patient Engagement Plan is a part of the Review and improve phase in the sub-criterion 3.1: Define and approve a methodology to engage stakeholders.

For patients, the most important benefit from the engagement in the R&I is its impact on their care and treatment: how they feel about their symptoms and/or functions.

Performance of patient engagement is about the success of your initiative in terms of participation. The associated metrics are to be selected ex-ante (before), included in the Patient Engagement Plan and verified ex-post (after) the development of the plan.

Core metrics	·        Number of different phases of the research process (Patient engagement in the Research & Innovation Path) patients were engaged in ·        Number of patients engaged across different socio-economic statuses, education backgrounds, genders, etc., to assess the capacity to engage diverse groups, including the most vulnerable ones ·        Number of engagement actions (online and offline) that took place, in which patients had an opportunity to express their views
Additional quantitative metrics	·        Number of KPIs selected to assess the impact of patient engagement ·        Number of training ·        Extent into which the patient involvement at the end is implemented in the research path ·        Number of interviews about patients’ experience in the engagement process ·        Number of co-created tools the engagement measurement ·        Number of reality test made by the patients ·        Number of patient 'intervention' directly, or indirectly
Additional qualitative metrics	·        Analysis of the patients’ expectation with respect to the patient engagement are met. ·        Analysis of whether the patients have felt engaged ·        Analysis of how meaningful the engagement was to the patients as well as to the research team. ·        Analysis of how patients have been engaged (e.g. collecting comments, surveys, feedback, etc.)

Table 7 Patient Engagement Performance Assessment metrics

Effectiveness of patient engagement is about success of your initiative in term of real impact of the participation on the research process: whether the actions performed have effectively produced impact and change in the R&I process. The metrics of effectiveness are included in the MULTI-ACT Digital Toolbox as a sub-set of the Patient Reported Dimension. You can also find the in the Appendix 4: Master Scorecard.

Core metrics	·        Number of changes in the research process (e.g. policies, composition of boards, objectives and priorities, strategic plan, evaluation of results, dissemination actions, etc.) according to the review made by patient. ·        Number of projects that include and show an effect on Patient Reported Outcomes (i.e. questionnaire reporting how they feel about symptoms and functions). ·        Number of projects involving patients in research activities, according to the needs of the mission.
Additional quantitative metrics	·        Number of changes in the research process (e.g. policies, composition of boards, objectives and priorities, strategic plan, evaluation of results, dissemination actions, etc.) according to the review made by patient. ·        Number of projects that include and show an effect on Patient Reported Outcomes (i.e. questionnaire reporting how they feel about symptoms and functions). ·        Number of projects involving patients in research activities, according to the needs of the mission. ·        Number of patients engaged in research activities, according to mission’s requirements. ·        The degree of representativeness: the number of the underrepresented population and of the disadvantaged patients involved in the research. ·        Number of dissemination actions carried out by patients (e.g. events where patients presented and endorsed research results). ·        Number of scientific articles in which patients are co-authors and/or reviewers. ·        Number of endorsements given by patients to research activities and results. ·        Number of endorsements given by patient organisations
Additional qualitative metrics	·        Analysis of whether patients’ expectation with respect to the research and mission of the initiative are met. ·        Analysis of the achievement in terms of new knowledge produced, from the perspective of all the stakeholders. ·        Analysis of how patients’ lives may be or have been improved by the research. ·        Analysis of the long-term improvement in health indicators ·        Analysis of whether the value of patient contribution if the same as others stakeholders ·        Evaluation of the project plan, of all single research phases and of the final results by patient and if and how their suggestions has been integrated into the research activities.

Table 8 Effectiveness and value assessment metrics

Co-Accountability: Materiality Analysis and Master Scorecard

 

This chapter will lead you through Materiality Analysis towards the final stage of creating your customised Master Scorecard, which your initiative will use to monitor and assess its progress towards the mission.

Understanding Materiality Analysis and Impact Assessment: important concepts

Below you will find explained some concepts which are crucial for this chapter. They are arranged in thematic groups rather than in alphabetical order to facilitate comprehension.

Co-accountability

Accountability means being responsible for what one does and being able to give a satisfactory reason for one’s actions. In the context of multi-stakeholder initiatives, accountability is a relationship among stakeholders who are required to give account for their actions: understanding them, reporting them, explaining them, accepting responsibility for their results.

Traditionally, accounting was addressed to shareholders and concentrated on financial results and processes. Emergence of environmental and social accounting marked broadening of scope of organizations’ responsibility, but also multiplication of perspectives to be taken into account. Nowadays, multiple categories of stakeholders both need to be consulted and reported to: not only shareholders, but stakeholders: customers, employees, local community, NGOs etc. In response to this shift in thinking, and even higher complexity of multi-stakeholder initiatives, MULTI-ACT puts forward the concept of co‑accountability: it is a democratic and participatory approach to implementing accountability, that incorporates plurality of stakeholder perspectives into decision-making processes, striving to build broad and holistic expert knowledge. Co-accountability is the theoretical foundation of the MULTI-ACT Framework: the five CRIF dimensions and the indicators they contain, make it possible for your initiative to establish priorities, monitor progress, report on the results and – last but not least – talk about your achievements in a language relevant to all key stakeholders. This way, participants of your initiative can communicate with one another and with the outside world about progress, results and the agenda.

Impact

Generally speaking, by “impact” we understand changes in the world (e.g. for the society, for patients) that happened as a result of an initiative’s activities. MULTI-ACT defines impact as long-term (over 5 years) socio-economic changes the intervention brings about, as opposed to “outcomes” which are more short-term.

Impact assessment

Impact assessment is a mean to measure the effects/changes/results that the initiative brings about. It includes conceptualization of the causal relationships between what inputs and impact, i.e. the research and other activities of an initiative and changes in the society (health improvement, higher well-being etc.). Additionally, some measure of both the activities and changes is needed. In the co-accountability framework, the indicators are not static but subject to change across time in order to properly respond to the changing environment and to the changing stakeholders needs.

Materiality

Not all changes brought about by an intervention or research initiative can be taken into account, and not all are equally relevant and significant. Choices have to be made about which data is tracked and reported, and when monitoring is optional: that is how impact indicators are chosen and created. Generally speaking, a piece of data is material if its omission or misrepresentation may affect stakeholders’ decisions or their ability to draw reasonable conclusions about the impact. It may be useful to think about materiality as a threshold above which missing or misrepresented information is considered to have an impact on the decision making of users.

Materiality Analysis

Naturally, organizations and individuals differ when deciding what is material, depending on their sector, mission, vision, values, strategy, dominant stakeholders and background.

In order to achieve co-accountability, stakeholders in your initiative participate in selection of the indicators. The process is called materiality analysis. Multi-Act makes it easy to conduct through the Collective Materiality Analysis in the Toolbox.

CRIF Dimensions

Collective Research Impact Framework created by MULTI-ACT project proposes a paradigm shift in health research impact assessment: from single stakeholder’s performance assessment perspective to a co-accountability model integrating multiple areas of impact assessment.

Therefore, conventional metrics of academic excellence in research were integrated with indicators of economic impact, financial balance, social influence and – last but not least – measures of capacity to accomplish one’s pre-defined mission. Patient-reported dimension overarches the other four dimensions, introducing the perspective of patient as the key stakeholder to impact assessment as a whole.

CRIF impact assessment comprises five dimensions that reflect different accountability perspectives: efficacy, excellence, economic, social and patient-reported that are described below.

·        Efficacy dimension

Efficacy dimension looks at your initiative’s capacity to accomplish the mission you defined for yourselves. This dimension is the main driver for co-accountability within CRIF, because it assesses to what extent the initiative brings value for its stakeholders as pre-defined in the mission. In the context of brain research, the mission focuses on the improvement of the life conditions of patients affected by brain diseases, while balancing the conflicting perspectives of the different stakeholders involved. This dimension has the strongest links to Governance Criteria.

·        Excellence dimension

This dimension focuses on the quality of scientific research that is conducted as part of you initiative. While it contains traditional bibliometric indicators used to measure academic performance, it goes beyond them, allowing for appraise contribution to knowledge and impact on society. The indicators reflect MULTI-ACT conviction that research should positively influence people’s lives to be deemed “excellent”.

·        Economic dimension

The economic dimension contains a set of economic and financial indicators. Monitoring your initiative’s internal financial balance is crucial to make it sustainable in the long run. Estimating its influence on the economy, e.g. through keeping patients in the workforce, is important for demonstrating its social impact in holistic manner.

·        Social dimension

In implementing social dimensions indicators, you are encouraged to look at the long-term direct and indirect effects of your initiative on the society as a whole, beyond primary stakeholders and beneficiaries. It also includes communication with the society (e.g. external reporting) and community engagement.

·        Patient-reported dimension (PRD)

Patient-reported dimension is the overarching one, in which the other four are rooted. It places patient at the centre of health research as the key stakeholder, whose needs and perspectives must be understood and incorporated into the research process. It is as tool enabling the Science of Patient Input since it includes indicators that are reported by patients. PRD comprises two types of indicators:

·        Patient-reported Outcomes (PROs)

·        Qualitative indicators to assess the Return on Engagement (RoE)

CRIF Aspects

Indicators within each dimension are grouped in order to make them more manageable, both at the stage of selection (Materiality Analysis) and later on, when you will implement them and use in monitoring your impact. They reflect assessment perspective.

Aspects are broader categories, and groups – more specific, narrower categories of indicators. In each aspect, there is at least one core indicator.

CRIF Indicators

The Master Scorecard provides a catalogue of 125 indicators, grouped into 5 dimensions.

Table 8 CRIF dimensions, aspects and indicators

Core and additional indicators

Not all indicators are equally important for each initiative. Core indicators are obligatory to use. There is at least one core indicator per aspect.

Table 9 Core and additional indicators

MULTI-ACT Framework recommends your initiative to select additional indicators that will increase its co-accountability in an aspect that you consider material, or in situations when your initiative is not able to apply a related core indicator, e.g. due to lack of required data. Indicators were selected and divided into these categories based on extensive literature review.

Qualitative and quantitative indicators

Indicators in the Master Scorecard, irrespective of their core or additional status, are of either qualitative or quantitative nature. This distinction will help you to figure out what kind of input is expected from you when you fill in each particular indicator in the Toolbox. For qualitative indicators, you are expected to provide a narrative description and/or mark your answer to a qualitative question. When it comes to quantitative indicator, some kind of numerical input is required.

Monitoring and evaluation framework[NM11] 

Regardless of whether an indicator is qualitative or quantitative, core or additional, it always pertains to one of the research stages. Research stages are derived from stages sometimes found in theory of change and impact assessment frameworks.

• Input indicators describe contributions made or required by stakeholders. They can include financial, human, technical and relational resources.
• Process indicators relate to the activities that make the research to happen (i.e. reviews, data collection, analysis, reporting).
• Outputs indicators represent tangible and intangible products resulting from research and innovation.
• Outcome indicators assess intermediate results and effects of the intervention (e.g. within 5 years since the intervention).
• Impact indicators, on the other hand, long-term socio-economic changes the intervention brings about (e.g. over 5 years since the intervention).

Figure 14 Monitoring and evaluation framework

Materiality Analysis

Within the Multi-Act Framework, materiality analysis is a process through which your initiative will determine which indicators it will be using for assessing its impact. Materiality analysis is a step towards co-accountability, as representatives of all stakeholders categories within your initiative will be engaged in selecting parameters that they will be collectively held responsible for monitoring and reporting. They will give their judgements on which aspects and indicators of the Master Scorecard are both:

·        relevant to your initiative’s mission and agenda and

·        significant enough to be considered material, i.e. their inclusion or omission may influence decision-making[NM12] 

Therefore, it is a necessary prerequisite to creating your customised Master Scorecard and to conducting impact assessment. Materiality analysis is a “bridge” leading from Governance to Impact Assessment part of the CRIF.

In the Materiality Analysis, you make use of information that you have already provided in the Baseline Analysis: mission and agenda and stakeholder engagement. If your initiative has not yet formulated mission and priorities, you need to do this before starting Materiality Analysis. Governance guidelines in the Criterion 1 will help you to do this. In the Criterion 2, you will find guidelines for effective stakeholder engagement.

It is your task, as the Promoter, to fill in a digital form in the Materiality Analysis Toolbox functionality to start the process. It requires mission and initial list of strategic priorities of your initiative with supporting documents[NM13] .

Promoters will be able to carry out materiality analysis via the Toolbox or other methods. Regardless of the means by which it is carried out (whether it is via questionnaire, online survey, interviews, etc.),there are some general recommendations to be followed to get a robust and reliable analysis:

1)     Cluster the responses by different stakeholder categories. Results can be then aggregated following the suggested methodology presented in the box below

2)     Ensure anonymity of the responses.

3)     Define a minimum number of individual views required to be considered representative of a stakeholder category (e.g. minimum 5), in order to ensure a balanced and veridical representation.

4)     Try to reach a heterogeneous cluster of responses within the same category: introducing additional specificities inside each stakeholder category helps catching potential differences within the same cluster (i.e. perspective of patients who are under treatment and not under treatment).

5)     Provide complete guidelines and/or tools to respondents that may not be fully aware of the initiative and the Multi-Act Framework.

6)     Clarify the threshold under which the responses are considered non-representative and thus inadmissible.

Scoring methodology

Premise Group all the results by category of stakeholder Focus on a single category of stakeholder (e.g. Care providers) Calculate the arithmetic average among the results of different stakeholders of the same category for each aspect. Replicate step 2 for each category of stakeholder. MATERIALITY: focus on aggregation of results of different stakeholder categories For each aspect, calculate the arithmetic average taking in consideration the result that the aspect obtained in every stakeholder category creating an “average of the averages”. (I.e. You should be calculating the average from the results that each category of stakeholder obtained in step 2) List the aspects taking in consideration their arithmetic average and sort them from the highest to the lowest. For each dimension, choose the top two aspect (i.e. for each dimension, the aspects featuring the highest and the second highest arithmetic average). Therefore, in this step you should select 10 aspects. Excluding the 10 aspects selected in step 5, choose additional 5 aspects with the highest arithmetic average. Combine results of step 5 and 6 to get to the 15 material aspects list. In case of the 15th aspect having the same average of 16th or following, then extend the selection to those.

 

Identification of stakeholders

The first step is to define which stakeholders will participate in the process. Identification of the stakeholders is of fundamental importance to conducting materiality analysis: it is the stage when you decide whose perspective will be included, who will co-create your goals and your assessment system. It is also a prerequisite to proper stakeholder engagement. Materiality analysis is a crucial part of the engagement, as stakeholders’ input is necessary for creating a balanced Master Scorecard that will capture your initiative’s mission and present its impact in a way meaningful for all stakeholders.

For each stakeholder category, which your initiative identified in the Baseline Analysis, you need to engage at least five representatives. This is important for ensuring a balanced voting process. They do not have to register in the Toolbox to participate in the Materiality Analysis through the Toolbox. For those unregistered, you need to add their e-mail addresses to the database under corresponding stakeholder category to allow them to participate.

Initiation

When you click “initiative MA”, an email is sent to all participants. It will contain a secure link to the MA interface in the Toolbox. Tokenization of the link guarantees:

·        anonymity of the participants.

·        association of every entry with its respective stakeholder category.

·        one-vote-per-participant rule.

[NM14] 

Figure 15 Materiality Analysis Tool admin interface

The Materiality Analysis Tool admin interface allows you to initiate the MA, monitor its results and conclude it. You can also use it to send reminders to the stakeholders.

Selection of aspects

The easiest way to engage stakeholders in materiality analysis is to invite them to participate in the process via Toolbox. You, as the Promoter, invite each stakeholder to select aspects and to indicate what are the most relevant aspects.

The participants will be requested to select CRIF aspects that are relevant and significant from their perspective. Each participant must select minimum one aspect from each of the five CRIF dimensions.

In the first step of aspect selection, the users are supplied with additional information to familiarize themselves with the listed aspects, as shown in Figure 2 – Materiality Analysis Tool Aspects & Grades .

Figure 16 Materiality Analysis Tool Aspects & Grades

 

Prioritization of aspects

Subsequently, each participant will be asked to prioritize the aspects they selected by grading them from 0 (non-relevant) to 5 (most relevant), separately for each dimension. At least two aspects in every dimension must be graded over 0.

Selection of indicators

After the final list of aspects is created, indicators under each aspect are added to the initiatives’ indicator list. Although it is advised that the final selection of indicators is limited to a manageable number: you can decide to limit the list to less than 15 aspects, or to expand it. The final set of selected indicators will constitute the customised Master Scorecard that you initiative will use to assess its impact and monitor its progress.

Once the materiality analysis is completed, relevant aspects should be defined. In the example below (Table 1 Example of a list of aspects), the materiality analysis rendered 12 relevant aspects. Number of relevant aspects can be lower than 15. Additionally, by using the Digital Toolkit, you can decide whether to expand or reduce the number of relevant aspects ranging from a minimum of XX to a maximum of XX.[NM15] 

MISSION	EXCELLENCE	ECONOMIC	SOCIAL	PATIENT REPORTED
Patient quality of life	Informing healthcare practice decision making	Improvement of health services	Political externalities	Quality of life
Drug supply to patient	-	-	-	Cognitive function
Health services and products accessibility	-	-	-	Locomotion
Stakeholder engagement	-	-	-	Fatigue
-	-	-	-	Patient satisfaction

Table 17 Example of a list of aspects

Identification of indicators

The materiality analysis leads to the identification of the indicators that your initiative will use to assess its outcomes and according to which it will report its performance. The Master Scorecard links every aspect to a number of indicators to be taken into account. For instance, since the initiative in the example above identified Patient quality of life as a relevant aspect, it should report on:

Aspect to be measured	Core/ Additional	Advantages
Patient quality of life	Core	The indicator measures the value of money of intervention since the quality-adjusted life year (QALY) is a generic measure of disease burden, including both the quality and the quantity of life.
Patient quality of life	Additional	The indicator measures the number of deaths.
Patient quality of life	Additional	The indicator measures the average time patient is expected to live.

Table 18 Patient quality of life: identification of indicators

Materiality analysis shows a snapshot in time of the stakeholders’ priorities; however, they may change over time. For this reason, materiality analysis should be carried out periodically, on a yearly or biyearly basis, in order to ensure its alignment with stakeholders’ priorities and their commitment to accomplishment of the initiative’s mission and agenda.

Only promoters can initially see intermediate results and monitor level of participation. When the participation prerequisites are met, the you can allow the MA results to be viewed by all the stakeholders. [NM16] When the analysis is complete, all participants are able to see the results as presented in Figure 3 – Materiality Analysis Results .

Figure 17 Materiality Analysis Results

The Toolbox presents the results of the materiality analysis in several ways (graphic, tabular etc.) to facilitate analysis.

Figure 18 Materiality Analysis Extended Results

Materiality analysis can be repeated at a future point in time, to better align with the overall developments and changes in the initiative. Past materiality analysis results are stored in the initiative’s database which enables further analysis.

Master Scorecard

WHAT is Master Scorecard?

The Master Scorecard is an adaptive tool for assessing health research & innovation initiatives and projects. It applies a multi-stakeholder perspective to provide a list of indicators for the assessment of research impact, considering the five CRIF dimensions (excellence, efficacy, social, economic and patient reported). The scorecard provides the indicators to evaluate the impact of health research and innovation, paying special attention to the benefits to patients, healthcare and society in line with the multi-stakeholder initiative’s mission. More specifically, its purpose is to offer innovative and multi-perspective approach for the organization to gather information on the achievement of its objectives concerning its impact .

It provides a catalogue of 125 indicators grouped into five CRIF dimensions. For each indicator, the scorecard offers its description, example, qualitative or quantitative measurement, and methods and data sources, among other details.

WHY using Master Scorecard and for what purposes?

The Master Scorecard enables strategic management of multi-stakeholder research initiatives. It assists initiatives in the evaluation of the multiple dimensions and impacts of their health research and innovation activities. It can be used as strategic management tool as it helps to monitor the progress of research and innovation projects and to demonstrate whether and how the initiatives are producing actual outcomes and impacts. However, the user must be aware that the computation of the indicators selected by an initiative from the scorecard will not provide an overall “score” or “ranking”.

MSC can be used in managing initiatives’ operations, identifying outcomes of the research, or for controlling and improving the initiative's performance.

The Master Scorecard is useful for initiatives aiming to increase the impact of research on people and society. The multi-stakeholder nature of MULTI-ACT allows the engagement of a broad range of users, which can be engaged in customizing and applying the Master Scorecard.

The Master Scorecard can be adapted to many individual needs:

• It allows flexibility and can be tailored to diverse multi-stakeholder projects, so it should not be used as a fixed set of indicators. It offers a starting point to be applied and tested in different contexts and settings, especially to multiple sclerosis or other brain diseases.
• It is dynamic as you can select indicators for different purposes and specific needs of many stakeholders through Materiality Analysis.
• It is constructed in a way that it can be used, customized and applied by a broad range of users. It is an indicator catalogue that covers a wide range of relevant aspects that can be used in assessing the multiple impacts of health research. Therefore, initiatives can select indicators among a different topics and possibilities according their  own needs.

HOW to use the Master Scorecard?

It can be applied at the beginning or during the development of a research initiative to select the indicators through the approach described in the Materiality Analysis. Depending on the stage of the project life cycle, the Master Scorecard can serve different purposes:

Initiation	Planning: The CRIF dimension (and the potential indicators) developed with the Master Scorecard allows the research initiative to (ex-ante) strategically design and evaluate the expected impact of a research project, according to its vision and agenda
Execution	Monitoring: the  Master Scorecard can serve to implement the mission selected by the initiative, in line with its vision and agenda. It can be used as a monitoring tool to assess the research and innovation activities delineated by CRIF dimensions. It could be used iteratively during the execution of the initiatives with the appropriate frequency
Closure	Assessment: the Master Scorecard can be applied at the end of the initiative in order to assess how the desired results were reached. If the Master Scorecard is applied from the beginning of project, the impacts can be compared with the initial evaluation output. This can help to strategically orient also future initiatives

Table 19 Master Scorecard use at different stages of initiative

After the completion of the Materiality Analysis, core indicators of the selected aspects are added to the initiative’s Master Scorecard. You may add other indicators to it. These additional indicators evaluate areas which are not covered by the core indicators but provide a more in-depth evaluation of the aspect.

Figure 20 Impact Assessment Indicators

Each indicator has an information page which describes how the indicator can be used for monitoring, when it is advisable to use it, and how to use it for measurement, how often to use it. You will also find links to external sources about each indicator. You can upload supporting documents for indicators, which allows you to keep things in order and keep track of the changes over the years.

Figure 21 Scorecard indicative Fiche – Results

Patient Reported Dimension

·        The impact assessment includes additional step for the Patient-Reported Dimension. To collect this extra data, you are encouraged to use the dedicated Toolbox functionality: the PRO Collection and Impact Analysis. It offers two ways to do so: questionnaires distributed via web surveys.

·        importing collective PRO results that had been exported from your in-house PRO database

The PRO data will be based on established questionnaires used by worldwide initiatives (such as iConquerMS [39], Italian MS Society PROMOPRO-MS[40]). The data will be used to calculate scores for your initiative’s performance. Using the PRO Data Collection and Impact Analysis functionality, you can:

·        Send a reminder to the participants that have not yet completed the survey.

·        View the overall participation percentage.

·        A historical depiction of the average scoring results per time period, both global and per individual question.

·        Export Reports

This Master Scorecard Tool’s administrative interface is accessible only to the promoters.

Figure 22 PRO Impact Analysis Interface

To start collecting PRO data through web surveys, you first need to upload participants’ e-mail addresses (either as a list or add one by one), and initiate the process. The Toolbox automatically sends an e-mail notifications containing links to the web questionnaire. Tokenization ensures that each participant can take the survey only once and the results will be collected anonymously. The web survey is hosted on the Toolbox. PRO data collection should be repeated periodically. The decision how often to do it is up to you. It is however recommended that you follow „Expected frequency” indicated on indicator information page.

Figure 23 ABILHAND Web Questionnaire

To import in-house, already collected PRO data, ensure that they have a designated format.

This format was defined using an export template that was provided by the PROMOPRO-MS.[NM17]  The data collected are not discarded unless you delete them.

 

 

 

 

 

Bibliography

 

Andreaus, Michele, and Ericka Costa, ‘Toward an Integrated Accountability Model for Nonprofit Organizations’, 2014, pp. 153–76 <https://doi.org/10.1108/S1041-706020140000017006>

Cabaj, Mark, and Liz Weaver, Collective Impact 3.0: An Evolving Framework for Community Change., 2016 <https://www.tamarackcommunity.ca/library/collective-impact-3.0-an-evolving-framework-for-community-change>

Caron-Flinterman, J. Francisca, Jacqueline E.W. Broerse, and Joske F.G. Bunders, ‘The Experiential Knowledge of Patients: A New Resource for Biomedical Research?’, Social Science & Medicine, 60 (2005), 2575–84 <https://doi.org/10.1016/j.socscimed.2004.11.023>

CML Advocates Network, ‘Community Advisory Board’, CML Advocates Network For Chronic Myeloid Leukemia Patient Group Advocates, 2018 <https://www.cmladvocates.net/cml-cab>

Cottam, Hilary, and Charles Leadbeater, Red Paper 01 Health: Co-CreatingServices, 2004 <https://www.designcouncil.org.uk/sites/default/files/asset/document/red-paper-health.pdf>

Cunningham‐Erves, Jennifer, Tilicia Mayo‐Gamble, Yolanda Vaughn, Jim Hawk, Mike Helms, Claudia Barajas, and others, ‘Engagement of Community Stakeholders to Develop a Framework to Guide Research Dissemination to Communities’, Health Expectations, 23 (2020), 958–68 <https://doi.org/10.1111/hex.13076>

Dhar, Vikrom K., Young Kim, Justin T. Graff, Andrew D. Jung, Jennifer Garrett, Lauren E. Dick, and others, ‘Benefit of Social Media on Patient Engagement and Satisfaction: Results of a 9-Month, Qualitative Pilot Study Using Facebook’, Surgery, 163 (2018), 565–70 <https://doi.org/10.1016/j.surg.2017.09.056>

Doria, Nicole, Brian Condran, Leah Boulos, Donna G. Curtis Maillet, Laura Dowling, and Adrian Levy, ‘Sharpening the Focus: Differentiating between Focus Groups for Patient Engagement vs. Qualitative Research’, Research Involvement and Engagement, 4 (2018), 19 <https://doi.org/10.1186/s40900-018-0102-6>

Engage2020, ‘Engage2020 Action Catalogue’, 2015 <http://actioncatalogue.eu/>

European Medicines Agency, EMEA Appendix 2 to the Guideline on the Evaluation of Anticancer Medicinal Products in Man, 2014, xliv

FDA, HHS, ‘Guidance for Industry Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims’, Clinical/Medical Federal Register, 2009, 1–39 <https://doi.org/10.1111/j.1524-4733.2009.00609.x>

Fontaine, Guillaume, Marc-André Maheu-Cadotte, Andréane Lavallée, Tanya Mailhot, Geneviève Rouleau, Julien Bouix-Picasso, and others, ‘Communicating Science in the Digital and Social Media Ecosystem: Scoping Review and Typology of Strategies Used by Health Scientists’, JMIR Public Health and Surveillance, 5 (2019), e14447 <https://doi.org/10.2196/14447>

Freeman, Edward R, Strategic Management : A Stakeholder Approach, Latest edi (Boston, MA, 1984)

Hall, Deborah Ann, Harriet Smith, Eithne Heffernan, and Kathryn Fackrell, ‘Recruiting and Retaining Participants in E-Delphi Surveys for Core Outcome Set Development: Evaluating the COMiT’ID Study’, ed. by Bridget Young, PLOS ONE, 13 (2018), e0201378 <https://doi.org/10.1371/journal.pone.0201378>

Hinrichs-Krapels, Saba, and Jonathan Grant, ‘Exploring the Effectiveness, Efficiency and Equity (3e’s) of Research and Research Impact Assessment’, Palgrave Communications, 2 (2016), 16090 <https://doi.org/10.1057/palcomms.2016.90>

Househ, Mowafa, Elizabeth Borycki, and Andre Kushniruk, ‘Empowering Patients through Social Media: The Benefits and Challenges’, Health Informatics Journal, 20 (2014), 50–58 <https://doi.org/10.1177/1460458213476969>

Humphrey-Murto, Susan, and Maarten de Wit, ‘The Delphi Method—More Research Please’, Journal of Clinical Epidemiology, 106 (2019), 136–39 <https://doi.org/10.1016/j.jclinepi.2018.10.011>

Joosten, Yvonne A., Tiffany L. Israel, Neely A. Williams, Leslie R. Boone, David G. Schlundt, Charles P. Mouton, and others, ‘Community Engagement Studios’, Academic Medicine, 90 (2015), 1646–50 <https://doi.org/10.1097/ACM.0000000000000794>

Kania, John, and Mark Kramer, ‘Collective Impact’, Stanford Social Innovation Review, Winter (2011) <https://ssir.org/images/articles/2011_WI_Feature_Kania.pdf>

Mazzucato, Maria, Mission-Oriented Research & Innovation in the European Union. A Problem-Solving Approach to Fuel Innovation-Led Grow, 2018 <https://doi.org/10.2777/36032>

Musso, Marta, Roberta Pinna, Giuseppe Melis, and Pier Paolo Carrus, ‘How Social Media Platform Can Support Value Cocreation Activities in Healthcare’, in Excellence in Services, 2018, pp. 535–55

OECD, ‘Glossary of Evaluation and Results Based Management (RBM) Terms’, 2010 <https://www.oecd.org/dac/evaluation/dcdndep/31950400.pdf>

Olesen, J., A. Gustavsson, M. Svensson, H.-U. Wittchen, and B. Jönsson, ‘The Economic Cost of Brain Disorders in Europe’, European Journal of Neurology, 19 (2012), 155–62 <https://doi.org/10.1111/j.1468-1331.2011.03590.x>

Schneeman, Kristin, Valerie Barton, and Brenda Huneycutt, Advancing Models of Patient Engagement: Patient Organizations as Research and Data Partners, 2019 <https://milkeninstitute.org/sites/default/files/reports-pdf/Full Series_Advancing Models of Patient Engagement- Patient Organizations as Research and Data Partners copy.pdf>

Strand, Roger, Jack Spaapen, Martin W Bauer, Ela Hogan, Gema Revuelta, Sigrid Stagl, and others, Indicators for Promoting and Monitoring Responsible Research and Innovation, 2015 <https://doi.org/10.2777/9742>

Thielst, Christina Beach, ‘Social Media: Ubiquitous Community and Patient Engagement.’, Frontiers of Health Services Management, 28 (2011), 3–14 <http://www.ncbi.nlm.nih.gov/pubmed/22256506>

WHO, Handbook for Guideline Development. 2nd Edition., 2014 <https://www.who.int/publications/guidelines/handbook_2nd_ed.pdf?ua=1>

de Wit, Maarten P.T., Tineke A. Abma, Marije S. Koelewijn-van Loon, Sarah Collins, and John Kirwan, ‘What Has Been the Effect on Trial Outcome Assessments of a Decade of Patient Participation in OMERACT?’, The Journal of Rheumatology, 41 (2013), 177–84 <https://doi.org/10.3899/jrheum.130816>

Wittchen, H.U., F. Jacobi, J. Rehm, A. Gustavsson, M. Svensson, B. Jönsson, and others, ‘The Size and Burden of Mental Disorders and Other Disorders of the Brain in Europe 2010’, European Neuropsychopharmacology, 21 (2011), 655–79 <https://doi.org/10.1016/j.euroneuro.2011.07.018>

Yaghmaei, Emad, ‘Responsible Research and Innovation Key Performance Indicators in Industry: A Case Study in the ICT Domain’, Journal of Information, Communication and Ethics in Society, 16 (2018), 214–34 <https://doi.org/10.1108/JICES-11-2017-0066>

 

 

 

Accronyms

ARSEP	Fondation Pour L'aide A La Recherche Sur La Slerose En Plaques
BA	Baseline Analysis
CC	Compliance Committee
CRIF	Collective Research Impact Framework
DiA	Dane-I-Analizy.Pl Sp Zoo
EBC	European Brain Council
EC	Europpean Commission
ECT	Engagement Coordination Team
EHMA	European Health Management Association
EU	European Union
EY SPA	Ernst & Young Financial Business Advisors
FISM	Fondazione Italiana Sclerosi Multipla Fism Onlus
INTRA	Intrasoft International
LB	Leadership Board
MA	Materiality Analysis
MSCU	Multiple Sclerosis Care Unit
PAB	Patient Advisory Board
PE	Patient Engagement
PRD	Patient-reported dimension
PROMs	Patient Reported Outcomes Measures
PROs	Patient Reported Outcomes
PwMS	People with multiple sclerosis
R&I	Research and Innovation
RFPO	Research Funding and Performing Organization
ROE	Return on Engagement
ROI	Return on Investment
RRI	Responsible Research & Innovation
SAB	Stakeholder Advisory Board
TAU	Tampereen Yliopisto
UBU	Universidad De Burgos
UCP	Universidade Catolica Portuguesa
UNITN	Universita Degli Studi Di Trento
WG	Committees and Working Groups

 

 

 

Glossary

Agenda: fundamental transformative objectives agreed upon by stakeholders that an initiative aims to achieve to fulfil its mission.

Appliers: RFPOs grouped in a multi-stakeholder initiative (e.g. Alliance) who implement the Multi-Act Framework.

Beneficiaries: individuals that benefit from the long-term direct or indirect effects of the initiative, which could be for example patients, their families and caregivers.

Breaking down the boundaries: see Research & Innovation Path.

Care providers: see Stakeholders.

Co-design: see Levels_of_Engagement.

Compliance Committee, CC: see Governance bodies.

Conduct & operate: see Research & Innovation Path.

Consult: see Levels_of_Engagement.

Collective Research Impact Framework, CRIF is a conceptual framework developed by Multi-Act enabling a new collective accountability approach to managing and assessment multi-stakeholder R&I initiatives.

CRIF Dimensions are a set of grouped indicators from Master Scorecard for assessing the impact of an initiative. For more, see CRIF Dimensions. There are five CRIF Dimensions defined below:

• Efficacy: refers to the capacity of a given initiative or programme to achieve its mission (strategic priorities set via the stakeholder engagement process). For more, see Efficacy dimension.
• Excellence: concerns the quality of research and its findings. For more, see Excellence dimension.
• Social: considers the direct and indirect effects of health research for the whole society, going beyond patient needs. For more, see Social dimension.
• Economic: refers to long-term financial sustainability of health R&I initiatives. For more, see Economic dimension.
• Patient-reported: concerns patients whose needs and perspectives must be understood and incorporated into health research impact evaluation. For more, see Patient-reported dimension (PRD).

Criteria and sub-criteria: a set of guiding principles that constitute the MULTI-ACT Governance Model and are intended to be followed by the Model's user.

Design & plan: see Research & Innovation Path.

Design and planning: see Research & Innovation Path.

Economic: see CRIF Dimensions.

Efficacy: see CRIF Dimensions.

Evaluating research: see Research & Innovation Path.

Excellence: see CRIF Dimensions.

Executing research: see Research & Innovation Path.

Experiential knowledge: knowledge gained through experience, as opposed to a priori (before experience) knowledge.

Framework: see Multi-stakeholder framework.

Governance bodies: groups with specific roles within a multi-stakeholder initiative that are composed by individuals participating to the initiative itself. For more, see Governance bodies.

• Leadership Board, LB: is composed by representatives from the categories of stakeholders that have a strategic importance for the initiative and represents the decision-making body. For more, see Leadership Board (LB).
• Stakeholder Advisory Board, SAB: a governance body composed by interested stakeholders and provides advices to the LB. Within this board, patients, their families and caregivers (one of the categories of stakeholders involved) might be asked by the LB to provide their specific contribution and advice for the most crucial decision-making processes according to the specific need of the initiative. This category of stakeholders can be defined as a sub-group within the SAB, called Patient Advisory Board (PAB).
• Committees and Working Groups (WG) can be appointed by the LB according to the specific needs of the program/project and the activities that will be carried out in order to achieve the desired change.
• Engagement Coordination Team (ECT) is in charge of coordinating the engagement of stakeholders, including patients, relatives and caregivers, in all the operations.
• Compliance Committee (CC) is in charge of maintaining a balance among stakeholders’ stances and expectations and oversee the ethical issues that might arise during the implementation of the initiative.

Governance Initiative: is a stage in multi-stakeholder initiative (including RFPOs) implementation process concerned with governance and management of a programme or a project, see Research & Innovation Path.

Health Research & Innovation, Health R&I: refers to “activities of research, technological development, demonstration and innovation, including the promotion of cooperation with non-EU countries and international organisations, the dissemination and optimisation of results and mobility of researchers in the Union” (Eur-lex, n.d.)[NM18]  within the healthcare domain.

Impact: is the long-term socio-economic changes the intervention brings about (e.g. over 5 years ).

Impact Indicator: is a "quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of a development actor" [41].

Industry: see Stakeholders.

Inform: see Levels_of_Engagement.

Initiative: see Multi-stakeholder initiative.

Input: the contributions made or required by each stakeholder/organization. It can include financial, human, technical and relational resources.

Involve: see Levels_of_Engagement.

Leadership Board, LB: see Governance_bodies.

Levels of Engagement: a way of describing the varying depth of patients’ involvement and agency in the research and innovation (R&I) process. For more, see Levels of Engagement.

• Co-design: Stakeholders are engaged with a decision-making role. For more, see Co-design.
• Involve: Stakeholders participate in research design and development as co-researchers and are engaged by providing their perspective and data. They are not involved in co-designing of the project as decision-makers. For more, see Involve.
• Consult: Stakeholders provide feedback for decision-making, give advice and opinions, but do not participate in decision-making. For more, see Consult.
• Inform: Stakeholders are informed about research in a passive role. For more, see Inform.

Mission: initiative’s current and future role, what it wants to achieve, and how it wants to achieve it.

Multi-stakeholder framework: is a conceptual structure applicable by/to a variety of stakeholders. Framework examples include (but are not limited to) guidelines, standards, certifications, normative schemes, etc.

Multi-stakeholder initiative: is a governance structure that seeks to bring different stakeholders together to participate in the dialogue, decision-making and implementation of solutions to the shared problems or goals.

Outcome: is the intermediate results and effects of the intervention (e.g. within 5 years), and is less tangible than outputs.

Output: is the activity in relation to each stakeholder’s inputs in quantitative terms. Alternatively, it can be defined as the tangible and intangible products resulting from research and innovation.

Patient Advisory Board, PAB: see Governance_bodies.

Patient-Provided Information: a range of input or data that is collected from patients.

Patient-Reported Outcomes, PROs: “any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else” [42].

Patient-Reported Outcomes Measures, PROMs: standardized, validated questionnaires (which are also called instruments) completed by patients to measure their perception of their functional well-being and health status.

Patient-Reported Dimension, PRD: see CRIF Dimensions.

Patients organizations: see Stakeholders.

Patients see Stakeholders.

Payers and purchasers: see Stakeholders.

Policy makers: see Stakeholders.

Process : “includes all the activities that enable the research to happen (i.e. reviewing of evidence, data collection, analysis, reporting and so forth)" [43].

Program Level: see Research & Innovation Path.

Project Level: see Research & Innovation Path.

Patient Engagement: is the action of engaging patients and their communities in R&I as key stakeholders with a decision-making role, “occurring when people with and affected by the disease meaningfully and actively collaborate in the governance, priority setting, and conduct of research, as well as in summarizing, distributing, sharing, and applying its resulting knowledge” [44].

R&I Path: see Research & Innovation Path.

R&I: see Health Research & Innovation.

Research Funding and Performing Organizations: see Stakeholders.

Research & Innovation Path (R&I Path): refers to sequence of processes and activities in the R&I where patients can be engaged in order to maximize the impact of R&I. For more, see Research & Innovation Path. Governance program level and project development levels are distinguished (also see Governance Initiative):

·        Program level: steps in multi-stakeholder initiative process concerned with the governance and management of research funding & performing programs:

o   Breaking down the boundaries: conditions that should be set in RFPOs in order to facilitate patient engagement as standard practice.

o   Setting research priorities: actions to establish justified interest in a specific research domain to a certain higher degree, importance, precedence, or rank over others.

o   Steering institutions: actions performed to establish steering and advisory committees and bodies.

o   Design and planning: the design and planning of all the activities that lead to the realization of a concept or idea and which helps achieve the item's designated objective(s).

o   Executing research: activities to actualize the research program or a specific research project for the purpose of achieving the item's designated objectives. Project level takes places at this stage.

o   Evaluating research: activities to determine the value created by a research program or project, establishing the outputs and outcomes, the degree to which the pre-established goals were achieved, and their impact.

o   Translation to community: activities to foster and facilitate the uptake of results of research programs or projects.

·        Project level: steps in multi-stakeholder initiative process concerned with performing single research projects. In this case, patient is a co-researcher. Project development pertains to research execution stage of the governance program level.

o   Conduct & operate: project conduct & monitoring (e.g. ICT device development).

o   Design & plan: the design and planning of all the activities that lead to the realization of a concept or idea and which helps achieve the designated objective(s).

o   Evaluation activities to determine the value created by a research project, establishing the outputs and outcomes, the degree to which the pre-established goals were achieved, and the impact.

o   Translation to community: activities to foster and facilitate the utilization? uptake of results of research projects.

Responsible Research and Innovation, RRI: research and innovation process in which societal actors work together in order to better align its outcomes with the values, needs and expectations of society (European Commission, n.d.).[NM19] 

Return on Engagement, RoE: the benefit, impact and value resulting from performing engagement in R&I. For more, see Return on Engagement (RoE).

Return on Investment, ROI: is a measure of the efficiency of an investment as a percentage of return relative to the investment’s cost.

RRI: see Responsible Research and Innovation.

Science of patient input: scientific research which uses data provided by people with a disease through passive or active contribution to evaluate its impact. For more, see Science of/with patient input

Science with patient input: scientific research where patients actively collaborate in governance, setting priorities, performance assessment etc. For more, see Science of/with patient input

Setting research priorities: see Research & Innovation Path.

Social Return On Investment, SROI: a method of measuring and accounting for extra-financial value (such as environmental or social value) for the stakeholders.

Social Dimension : see CRIF_Dimensions.

Society: see Stakeholders.

Stakeholder: refers to “any individual or group that is affected by, who can influence or may have an interest in the outcomes of an organization’s actions” [45].

• Patients: people with the disease (persons with lived experience of the disease); and people affected by the disease (persons or groups that are affected by the disease, including family members and caregivers).
• Patients organizations: not-for profit organisations which are patient focused, where patients and/or their carers constitute majority in governing bodies, e.g. patient associations, advocacy organizations.
• Society: individual citizens, civil society organizations and networks.
• Payers and purchasers: public or private entities responsible for underwriting the costs of health care.
• Care providers: health and social care organizations and professionals (doctors, nurses, etc.).
• Policy makers: EU institutions; national, regional and local policy makers of different levels.
• Regulators: regulatory agencies and Health Technology Assessment (HTA) bodies, e.g. agencies for the scientific evaluation and safety monitoring of medicines, i.e. the European Medicine Agency.
• Industry: companies developing and selling health products and services.
• Research Funding and Performing Organizations, RFPO: universities, research hospitals, research projects, foundations, and all private and public research funders.

Promoter: individuals that guide the adoption of Multi-Act within their organizations or initiatives, and are members of their governance bodies.

Stakeholder Advisory Board, SAB: see Governance_bodies.

Steering institutions: see Research & Innovation Path.

Sub-criterion: see Criteria.

Transformational mission: mission of research initiative that shifts or breaks existing scientific paradigms.

Translation to community: see Research & Innovation Path.