Performance Evaluation of National R&D in the Field of ...
2008 AEA’s annual conference
Performance Evaluation of National R&D in the Field of Environmental Technology with respect to Science and Technology Classification
Noeon Park, HyunJeong Cho, Seung Jun Yoo, Yule Shin, MoonJung Choi, Sang-youb Lee Korea Institute of S&T Evaluation and Planning (KISTEP)
2/18
I . INTRODUCTION II . OBJECTIVE III . METHODOLOGY IV . RESULTS Ⅴ.
CONCLUSIONS
3/18
INTRODUCTION Environmental Sustainability Index (ESI) Korea ranked 122nd among 146 countries, in 2005 No. Indicators
ESI Ranking
Env. Systems
4
18
Reducing Env. Stresses
3
78
Reducing Human Vulnerability
5
137
Social & Institutional Capacity
3
67
Global Stewardship
6
146
Total
21
122
Environmental Sustainability Index, 2005
4/18
INTRODUCTION R&D investment analysis Total budget(‘03~’07) : $ 16 billion (74% by government) Ministry of Commerce(33%) Ministry of Environment (31%) Ministry of Science and Technology (20%)
R&D performance analysis (Number/$ 70 thousand)
2003
2004
2005
2006
2007
SCI Paper
0.22
0.27
0.32
0.28
0.26
Patent
0.21
0.18
0.20
0.23
0.20
National R&D (‘06) : 0.17(SCI paper), 0.10(patent)
INTRODUCTION S&T Classification system National S&T standard classification system It is based on Science and Technology Basic Law, 2002 It has periodically been updated to improve the trend of global R&D programs every 3 year It is classified by 3-digit (19 divisions, 160 disciplines, 1,023 subjects)
Institutional classification system It is provided by Ministry of Environment, Korea It has been used for ECO-TECHNOPIA 21 PROGRAM(‘01~’10) It is clssified by 2-digit (12disciplines, 30 subjects)
Others (6T, NTRM)
5/18
OBJECTIVE
6/18
Motivation In-depth Evaluation of R&D Program It has been started from 2008 in Korea R&D programs can be represented by performance which are generally produced in the forms of papers, patents, technological royalty, etc. It has generally been evaluated by summation method with respect to efficiency, effectiveness, feasibility, and accountability
Objective In-depth Evaluation of R&D program using classification system In this study, the effectiveness of R&D program was investigated using new classification system in the field of environment technology
METHODOLOGY
7/18
Selection of Program ECO-TECHNOPIA 21 Aim : Building an environmentally healthy country where the ecology and the economy support each other, by developing environmental technologies extensively Program Period : 2001 ~ 2010 (10 years) Total Budget : $ 695 thousand
Performance Evaluation ('06)
In-depth evaluation using new classification system Categories for performance evaluation were rearranged into 8 disciplines based on classification system provided by Ministry of Environment It is evaluated in terms of papers (SCI, domestic), patents (international, domestic) and technological royalty
8/18
METHODOLOGY
Investment analysis for ECO-TECHNOPIA 21 program ('06)
(a) By R&D types
(b) By performer types
9/18
METHODOLOGY Major performance Indicators for ECO-TECHNOPIA 21 Indicators Technological royalty No. SCI paper No. providing of DB guide manual No. domestic patent Sales amount Self-evaluation(‘07)
10/18
RESULTS Classification system in the field of ET Ministry of Environment(12)
National S&T classification(8)
Clean and safe air Eco-friendly materials
Pollution prevention and clean elements
Eco-friendly process
Pollution control and management
Soil∙ groundwater restoration∙management Ecological restoration∙management Clean drinking water
Environmentally friendly materials, parts and facilities
Wastewater treatment
Environmental preservation and remediation
Waste management∙ recycling
Environmental health
Risk evaluation Measurement apparatus Comfortable life Global issue (climate change)
Environmental information Environmental prediction, monitoring and evaluation Prediction and mitigation of environmental and natural hazards
11/18
RESULTS New classification system Categories(8)
Extent
Eco-friendly industry
Materials, Process, System
Air environment
Materials, Treatment, Evaluation
Water environment
Materials, Treatment, Evaluation
Soil∙ groundwater environment
Materials, Treatment, Restoration, Evaluation
Ecology
Restoration, Evaluation
Environmental health
Evaluation
Waste management∙ recycling
Treatment, Recycling
others
Noise evaluation, Education
12/18
RESULTS Performance Data ('06) Project start year 2006
SCI paper
(Unit : number, %)
Domestic paper
No. No. No. project outputs project 46(65% 120(74% 80(71%) ) )
No. outputs 230(77%)
International patent
Domestic patent application
No. project 12(71% )
No. outputs 18(58% )
No. project 30(68% )
No. outputs 35(67% )
2005
15
27
27
47
1
1
11
14
2004
7
5
12
18
2
6
2
2
2003
3
1
2
3
2
6
1
1
2002
0
0
1
0
0
0
0
0
Total
71
113
162
298
17
31
44
52
13/18
RESULTS Performance Data ('06)
(Unit : %)
2006
SCI paper
Domestic paper
Int’l patent
Domestic patent
Basic
33
25
1
5
Applied
47
44
45
65
Development
20
31
54
30
Total
100
100
100
100
2006
Basic research in performance Paper patent
14/18
RESULTS Performance Data ('06)
(Unit : Number, $ million) Project start year(‘02~’06)
SCI paper
Domestic paper
Int’l patent
Domestic patent
Tech. royalty
Eco-friendly industry
55
32
1
3
0.5
Air environment
12
67
5
4
0.7
Water environment
17
45
8
25
0.9
Soil∙ groundwater environment
2
17
0
1
0.4
Ecology
14
63
0
3
0.2
Environmental health
17
27
0
1
0.2
Waste management∙ recycling
0
19
4
12
0.6
others
24
29
12
1
0.5
15/18
RESULTS Performance Data ('06)
(Unit : $ million, %) Project started year(‘06)
Total funding(‘06)
SCI paper1)
Domestic paper1)
Int’l patent1)
Domestic patent1)
Eco-friendly industry
7.8
14.5
24.7
0.9
1.7
Air environment
12.2
5.1
39.3
7.5
5.4
Water environment
15.4
11.2
49.1
14.5
31.3
Soil∙ groundwater environment
2.9
5.7
25.6
0
0
Ecology
5.1
22.4
63.0
0
1.8
Environmental health
3.6
31.1
50.7
0
4.3
Waste management∙ recycling
7.0
0
24.9
7.4
15.9
others
13.9
8.9
15.3
0.4
0.9
1)The
value means the ratio of funding between outputs and total at each area
16/18
RESULTS Performance Data ('06) Water environment(69)
Ratio Relative tech. level Ecology(58) Eco-friendly industry(60)
CONCLUSIONS
17/18
New classification system is provided to evaluate R&D program in the field of environmental technology. Outputs, including papers, patents and royalty, were mainly produced through applied and development researches. Many papers were published in the field with relatively low technology level including ecology and eco-friendly industry. Many patents were published in the field with relatively high technology level including water environment.
FURTHER RESEARCH
18/18
Performance evaluation with same classification system in 2007 Comparative evaluation with similar program of different ministry Development of new tools for qualitative evaluation Suggestion of new classification system in the field of environment technology in detail
Performance Evaluation of National R&D in the Field of Environmental Technology with respect to Science and Technology Classification
Noeon Park, HyunJeong Cho, Seung Jun Yoo, Yule Shin, MoonJung Choi, Sang-youb Lee Korea Institute of S&T Evaluation and Planning (KISTEP)
2/18
I . INTRODUCTION II . OBJECTIVE III . METHODOLOGY IV . RESULTS Ⅴ.
CONCLUSIONS
3/18
INTRODUCTION Environmental Sustainability Index (ESI) Korea ranked 122nd among 146 countries, in 2005 No. Indicators
ESI Ranking
Env. Systems
4
18
Reducing Env. Stresses
3
78
Reducing Human Vulnerability
5
137
Social & Institutional Capacity
3
67
Global Stewardship
6
146
Total
21
122
Environmental Sustainability Index, 2005
4/18
INTRODUCTION R&D investment analysis Total budget(‘03~’07) : $ 16 billion (74% by government) Ministry of Commerce(33%) Ministry of Environment (31%) Ministry of Science and Technology (20%)
R&D performance analysis (Number/$ 70 thousand)
2003
2004
2005
2006
2007
SCI Paper
0.22
0.27
0.32
0.28
0.26
Patent
0.21
0.18
0.20
0.23
0.20
National R&D (‘06) : 0.17(SCI paper), 0.10(patent)
INTRODUCTION S&T Classification system National S&T standard classification system It is based on Science and Technology Basic Law, 2002 It has periodically been updated to improve the trend of global R&D programs every 3 year It is classified by 3-digit (19 divisions, 160 disciplines, 1,023 subjects)
Institutional classification system It is provided by Ministry of Environment, Korea It has been used for ECO-TECHNOPIA 21 PROGRAM(‘01~’10) It is clssified by 2-digit (12disciplines, 30 subjects)
Others (6T, NTRM)
5/18
OBJECTIVE
6/18
Motivation In-depth Evaluation of R&D Program It has been started from 2008 in Korea R&D programs can be represented by performance which are generally produced in the forms of papers, patents, technological royalty, etc. It has generally been evaluated by summation method with respect to efficiency, effectiveness, feasibility, and accountability
Objective In-depth Evaluation of R&D program using classification system In this study, the effectiveness of R&D program was investigated using new classification system in the field of environment technology
METHODOLOGY
7/18
Selection of Program ECO-TECHNOPIA 21 Aim : Building an environmentally healthy country where the ecology and the economy support each other, by developing environmental technologies extensively Program Period : 2001 ~ 2010 (10 years) Total Budget : $ 695 thousand
Performance Evaluation ('06)
In-depth evaluation using new classification system Categories for performance evaluation were rearranged into 8 disciplines based on classification system provided by Ministry of Environment It is evaluated in terms of papers (SCI, domestic), patents (international, domestic) and technological royalty
8/18
METHODOLOGY
Investment analysis for ECO-TECHNOPIA 21 program ('06)
(a) By R&D types
(b) By performer types
9/18
METHODOLOGY Major performance Indicators for ECO-TECHNOPIA 21 Indicators Technological royalty No. SCI paper No. providing of DB guide manual No. domestic patent Sales amount Self-evaluation(‘07)
10/18
RESULTS Classification system in the field of ET Ministry of Environment(12)
National S&T classification(8)
Clean and safe air Eco-friendly materials
Pollution prevention and clean elements
Eco-friendly process
Pollution control and management
Soil∙ groundwater restoration∙management Ecological restoration∙management Clean drinking water
Environmentally friendly materials, parts and facilities
Wastewater treatment
Environmental preservation and remediation
Waste management∙ recycling
Environmental health
Risk evaluation Measurement apparatus Comfortable life Global issue (climate change)
Environmental information Environmental prediction, monitoring and evaluation Prediction and mitigation of environmental and natural hazards
11/18
RESULTS New classification system Categories(8)
Extent
Eco-friendly industry
Materials, Process, System
Air environment
Materials, Treatment, Evaluation
Water environment
Materials, Treatment, Evaluation
Soil∙ groundwater environment
Materials, Treatment, Restoration, Evaluation
Ecology
Restoration, Evaluation
Environmental health
Evaluation
Waste management∙ recycling
Treatment, Recycling
others
Noise evaluation, Education
12/18
RESULTS Performance Data ('06) Project start year 2006
SCI paper
(Unit : number, %)
Domestic paper
No. No. No. project outputs project 46(65% 120(74% 80(71%) ) )
No. outputs 230(77%)
International patent
Domestic patent application
No. project 12(71% )
No. outputs 18(58% )
No. project 30(68% )
No. outputs 35(67% )
2005
15
27
27
47
1
1
11
14
2004
7
5
12
18
2
6
2
2
2003
3
1
2
3
2
6
1
1
2002
0
0
1
0
0
0
0
0
Total
71
113
162
298
17
31
44
52
13/18
RESULTS Performance Data ('06)
(Unit : %)
2006
SCI paper
Domestic paper
Int’l patent
Domestic patent
Basic
33
25
1
5
Applied
47
44
45
65
Development
20
31
54
30
Total
100
100
100
100
2006
Basic research in performance Paper patent
14/18
RESULTS Performance Data ('06)
(Unit : Number, $ million) Project start year(‘02~’06)
SCI paper
Domestic paper
Int’l patent
Domestic patent
Tech. royalty
Eco-friendly industry
55
32
1
3
0.5
Air environment
12
67
5
4
0.7
Water environment
17
45
8
25
0.9
Soil∙ groundwater environment
2
17
0
1
0.4
Ecology
14
63
0
3
0.2
Environmental health
17
27
0
1
0.2
Waste management∙ recycling
0
19
4
12
0.6
others
24
29
12
1
0.5
15/18
RESULTS Performance Data ('06)
(Unit : $ million, %) Project started year(‘06)
Total funding(‘06)
SCI paper1)
Domestic paper1)
Int’l patent1)
Domestic patent1)
Eco-friendly industry
7.8
14.5
24.7
0.9
1.7
Air environment
12.2
5.1
39.3
7.5
5.4
Water environment
15.4
11.2
49.1
14.5
31.3
Soil∙ groundwater environment
2.9
5.7
25.6
0
0
Ecology
5.1
22.4
63.0
0
1.8
Environmental health
3.6
31.1
50.7
0
4.3
Waste management∙ recycling
7.0
0
24.9
7.4
15.9
others
13.9
8.9
15.3
0.4
0.9
1)The
value means the ratio of funding between outputs and total at each area
16/18
RESULTS Performance Data ('06) Water environment(69)
Ratio Relative tech. level Ecology(58) Eco-friendly industry(60)
CONCLUSIONS
17/18
New classification system is provided to evaluate R&D program in the field of environmental technology. Outputs, including papers, patents and royalty, were mainly produced through applied and development researches. Many papers were published in the field with relatively low technology level including ecology and eco-friendly industry. Many patents were published in the field with relatively high technology level including water environment.
FURTHER RESEARCH
18/18
Performance evaluation with same classification system in 2007 Comparative evaluation with similar program of different ministry Development of new tools for qualitative evaluation Suggestion of new classification system in the field of environment technology in detail
