Wim van Dalen Assessment Criteria
11/30/16
RARHA WP6: Assessment Criteria, procedure and results Wim van Dalen Kirsten Vegt Work Package 6 Satellite Symposium , 21st November 2016
Basis: Assessment system of the RIVM
RIVM: Dutch Institute for Public Health and the Environment; Ministry of Health, Welfare and Sport
1
11/30/16
Working principles assessment STAP q Goal: to gather specific information and concrete examples of good practice on which some evidence of effectiveness and cost estimates are available, with special attention on good practice in information dissemination. q Purpose and methods: Partners are invited to provide examples of three group of interventions q q q
Early intervention services (including brief advices) School-based programs (information and education) Public awareness programs (including new media, social networks and online tools for behavior change).
Assessment criteria in practice
q The intervention is well-described q The intervention is implemented in the real world/ feasible/transferable q The intervention has a theoretical base q The intervention has been evaluated
2
11/30/16
Results Early Public School Total interventions Awareness Based Interventions Interventions Rejected Interventions
10
3
5
18
Accepted interventions
11
7
8
26
Total # interventions received
21
10
13
44
% Accepted
52%
70%
62%
59%
Assessment criteria Level of evidence •
Basic level: theoretically sound and with positive results (observational or qualitative studies)
•
First indications for effectiveness (pre- and post-design)
•
Good indications for effectiveness (pre-post controlled design)
•
Strong indications for effectiveness (pre-post controlled design with follow-up)
6
3
11/30/16
Results Accepted Interventions Level of Evidence
Early interventions
Public Awareness Interventions
School Based Interventions
Total
Basic Level
4
4
0
8
First indications for effectiveness
1
2
2
5
Good indications for effectiveness
1
1
3
5
Strong indications for effectiveness
5
0
3
8
Total
11
7
8
26
What about rejected interventions (1)
•
Common requirements that weren’t met during assessment: q The intervention is well-described: A big problem that would often arise during assessment was that the goal of the intervention wasn’t clearly described. Furthermore, the description of the intervention was often not complete or clear. For example, and intervention would have information on frequency, but not on duration. q The intervention is implemented in the real world/ feasible/transferable: Specifics on financial costs or time that needed to be invested were often missing or unclear, as well as that there wasn’t a manual or a concrete description of activities for the intervention available.
4
11/30/16
What about rejected interventions (2)
•
Common requirements that weren’t met during assessment: q The intervention has a theoretical base: It was often the case that there weren’t any effective elements (or techniques or principles) in the approach stated or specified, in the framework of a change model or an intervention theory, or based on results of previously conducted research. q The intervention has been evaluated: The outcomes found weren’t always the most relevant given the objective that was stated in the intervention description. This often occurred simultaneously with an unclear description of the intervention goal. In these cases, it was impossible to assess the effectiveness of the intervention properly.
Results per Country Country
Austria Bulgaria Croatia Cyprus Finland Germany Greece Ireland Italy Lithuania Luxembourg Netherlands Norway Poland Portugal Slovenia Spain Sweden Total
Submitted interventions
3 1 2 1 2 2 2 2 2 2 1 2 3 2 5 3 2 7 44
Accepted interventions
1 2 2 1 1 1 2 1 1 2 2 2 2 2 1 3 26
(Of which reassessed)
1 2 2 1 1 1 1 1 2 1 1 1 15
Rejected interventions
2 1 1 1 1 1 1 1 3 1 1 4 18
(Request for more information was made, none received) 1 1 1 1 3 7
5
11/30/16
Thanks Wim van Dalen
Dutch Institute for Alcohol Policy (STAP) Postbus 9769 | 3506 GT | Utrecht | The Netherlands T: +31 (0)30 6565 041 | M: +31 (0) 6 53295544 E: [email protected] | twitter: @wimvandalen I: www.stap.nl •
6
RARHA WP6: Assessment Criteria, procedure and results Wim van Dalen Kirsten Vegt Work Package 6 Satellite Symposium , 21st November 2016
Basis: Assessment system of the RIVM
RIVM: Dutch Institute for Public Health and the Environment; Ministry of Health, Welfare and Sport
1
11/30/16
Working principles assessment STAP q Goal: to gather specific information and concrete examples of good practice on which some evidence of effectiveness and cost estimates are available, with special attention on good practice in information dissemination. q Purpose and methods: Partners are invited to provide examples of three group of interventions q q q
Early intervention services (including brief advices) School-based programs (information and education) Public awareness programs (including new media, social networks and online tools for behavior change).
Assessment criteria in practice
q The intervention is well-described q The intervention is implemented in the real world/ feasible/transferable q The intervention has a theoretical base q The intervention has been evaluated
2
11/30/16
Results Early Public School Total interventions Awareness Based Interventions Interventions Rejected Interventions
10
3
5
18
Accepted interventions
11
7
8
26
Total # interventions received
21
10
13
44
% Accepted
52%
70%
62%
59%
Assessment criteria Level of evidence •
Basic level: theoretically sound and with positive results (observational or qualitative studies)
•
First indications for effectiveness (pre- and post-design)
•
Good indications for effectiveness (pre-post controlled design)
•
Strong indications for effectiveness (pre-post controlled design with follow-up)
6
3
11/30/16
Results Accepted Interventions Level of Evidence
Early interventions
Public Awareness Interventions
School Based Interventions
Total
Basic Level
4
4
0
8
First indications for effectiveness
1
2
2
5
Good indications for effectiveness
1
1
3
5
Strong indications for effectiveness
5
0
3
8
Total
11
7
8
26
What about rejected interventions (1)
•
Common requirements that weren’t met during assessment: q The intervention is well-described: A big problem that would often arise during assessment was that the goal of the intervention wasn’t clearly described. Furthermore, the description of the intervention was often not complete or clear. For example, and intervention would have information on frequency, but not on duration. q The intervention is implemented in the real world/ feasible/transferable: Specifics on financial costs or time that needed to be invested were often missing or unclear, as well as that there wasn’t a manual or a concrete description of activities for the intervention available.
4
11/30/16
What about rejected interventions (2)
•
Common requirements that weren’t met during assessment: q The intervention has a theoretical base: It was often the case that there weren’t any effective elements (or techniques or principles) in the approach stated or specified, in the framework of a change model or an intervention theory, or based on results of previously conducted research. q The intervention has been evaluated: The outcomes found weren’t always the most relevant given the objective that was stated in the intervention description. This often occurred simultaneously with an unclear description of the intervention goal. In these cases, it was impossible to assess the effectiveness of the intervention properly.
Results per Country Country
Austria Bulgaria Croatia Cyprus Finland Germany Greece Ireland Italy Lithuania Luxembourg Netherlands Norway Poland Portugal Slovenia Spain Sweden Total
Submitted interventions
3 1 2 1 2 2 2 2 2 2 1 2 3 2 5 3 2 7 44
Accepted interventions
1 2 2 1 1 1 2 1 1 2 2 2 2 2 1 3 26
(Of which reassessed)
1 2 2 1 1 1 1 1 2 1 1 1 15
Rejected interventions
2 1 1 1 1 1 1 1 3 1 1 4 18
(Request for more information was made, none received) 1 1 1 1 3 7
5
11/30/16
Thanks Wim van Dalen
Dutch Institute for Alcohol Policy (STAP) Postbus 9769 | 3506 GT | Utrecht | The Netherlands T: +31 (0)30 6565 041 | M: +31 (0) 6 53295544 E: [email protected] | twitter: @wimvandalen I: www.stap.nl •
6
