Past, Present, and Possible Futures
Behavioral and Experimental Economics and USDA Conservation Policy: Past, Present, and Possible Futures
Marca Weinberg Division Director, RRED/ERS/USDA Conference on Behavioral & Experimental Agri-environmental Research: Methodological Advancements & Applications to Policy Shepherdstown, WV; October 14, 2017
“Richard Thaler won the Nobel Prize for making economics more human — and more real” Washington Post, October 9 2017
2
Evidence-based Policy is an Overarching Objective
The President recently asked his Cabinet to carry out an aggressive management agenda for his second term that delivers a smarter, more innovative, and more accountable government for citizens. An important component of that effort is strengthening agencies' abilities to continually improve program performance by applying existing evidence about what works, generating new knowledge, and using experimentation and innovation to test new approaches to program delivery. OMB M-13-17 July 26 2013
Evidence and Evaluation. The Administration is committed to building evidence and better integrating evidence into policy, planning, budget, operational, and management decision-making. We appreciate agencies’ efforts to strengthen the use of data and evidence to drive better decision making and achieve greater impact and efficiencies. To further these efforts, agencies should submit: 1) proposals to advance agency efforts to build and use a portfolio of evidence; and 2) proposals to strengthen agency infrastructure and capacity to use evidence, evaluation, and data as tools to improve Federal Government effectiveness. OMB M-17-28 July 7, 2017
3
Evidence-based Policy is an Overarching Objective
Evidence-Based Policymaking Commission Act of 2016 • The Promise of Evidence-Based Policymaking Final Report Released September 7, 2017
• Focus is on the use of Administrative Data – Lower costs – Reduce respondent burden – Many concepts apply to policy experiments
4
USDA Conservation Programs Seek to Improve Environmental Performance in Agriculture A portfolio of programs ….. Conservation Reserve Program (CRP) Agricultural Conservation Easement Program (ACEP)
Land Retirement/Preservation Programs • •
Environmental Quality Incentives Program (EQIP) Conservation Stewardship Program (CSP)
Working Lands Programs • •
Conservation Technical Assistance (CTA)
With multiple objectives…. Protect and improve water quality and quantity, soil health, wildlife habitat, air quality, and more
5
USDA Conservation Programs Seek to Improve Environmental Performance in Agriculture
6
All Farmers
Program Applicants
Participation Incentive What action? • Fields • Practices
All Eligible Farmers
Program Budget
Program Participants
Program Performance • Economic • Environmental • Distributional
A multitude of program design decisions affect conservation program outcomes
Eligibility • Land types • Land use • Location • Practices
What payment? • Fixed • Bid
Enrollment Screen (targeting) • Benefit-cost index • Field Location –Soils –Topography –Population • Practices • Cost (bid)
7
ERS has a long history doing, and funding, experimental work Nutrition (BEN) Agri-environmental Policy (CBEAR) Food Assistance (BECR)
Competitive Grants to Create Centers • • •
FSA CRP Nudge beginning in 2011 Behavioral Economics Master Class (2014 with Ideas 42)
Partnerships with Program Agencies • •
ERS Research and Collaboration
• Cooperative agreements (many we could list here) • ERS researcher projects and publications (e.g. 2017 Hellerstein et al ERR)
8
Economic Experiments for Policy Analysis and Program Design: A Guide for Agricultural Decisionmakers
Higgins, et al. 2017. ERS Economic Research Report 236. https://www.ers.usda.gov/publications/pub-details/?pubid=84668
9
Types of Economic Experiments
10
11
Next Steps: More Ambitious Field Experiments? Much of ERS’ field work has been on “Outreach Experiments”
• Advantages: Low cost to implement, administrative data as outcome measures, a “foot in the door” for field experiments. • Cons: Generally small treatment effects; don’t address core program design questions. • Examples: CRP Outreach, Riparian Buffers, Organic Soils (non-environ: Microloans, FSA county elections).
A more ambitious goal: Focus on program design • • • •
Examples: Auction structure, Defaults, Payment levels, Contract design. Advantages: Addresses core program design questions. Motivations: Programs are often already “trying different things.” Obstacles: More difficult to design, very difficult to get approvals, no dedicated evaluation funding (e.g. DOE).
12
• • • •
Challenges
Budget issues: Experiments are expensive and budgets are uncertain It is always a challenge, and time consuming, to work with bureaucracy (e.g., OMB implementation of the Paper Work Reduction Act) Program agencies may be constrained by mandate, regulation, capacity and culture, and concerned about privacy, equity, respondent burden, public opinion, etc. The traditional (ex-post) peer-review process is not optimized for reviewing experiments
– Assumptions underlying experimental research are imposed ex-ante, as part of the design process – Revising and repeating experiments may be prohibitively expensive
13
ERS is Adopting a Formal Process for Approving Experimental Research We designed a multi-phase process with four “gates”*
1. Specify research objective & approach, develop a timeline, and secure collaborator commitments 2. Peer review of Experimental Design and Analysis Plan -- recruitment plan, protocols & instruction, randomization protocols, modeling strategies, etc. 3. Formal administrative approvals – formal collaborator commitments, OMB review, IRB clearance 4. Implementation [5. Analysis and reporting; with the plans from above serving to discipline and justify the analytical approach]
* Following templates created by GSA’s (now defunct) Social Behavioral Science Team
14
Final Thoughts: How can (non-USDA) research make a difference?
• Know your question! • Know the decision context of individuals AND program managers • Ensure your work can withstand critical scrutiny (power tests, mechanism commitment, etc.) • Validity – field tests (including administrative experiments) are always attractive • MANY interesting questions remain unanswered… – Do defaults matters (say, when land retirement offers are solicited) – New auction mechanisms – Self monitoring & performance-based incentives
15
Marca Weinberg Division Director, RRED/ERS/USDA Conference on Behavioral & Experimental Agri-environmental Research: Methodological Advancements & Applications to Policy Shepherdstown, WV; October 14, 2017
“Richard Thaler won the Nobel Prize for making economics more human — and more real” Washington Post, October 9 2017
2
Evidence-based Policy is an Overarching Objective
The President recently asked his Cabinet to carry out an aggressive management agenda for his second term that delivers a smarter, more innovative, and more accountable government for citizens. An important component of that effort is strengthening agencies' abilities to continually improve program performance by applying existing evidence about what works, generating new knowledge, and using experimentation and innovation to test new approaches to program delivery. OMB M-13-17 July 26 2013
Evidence and Evaluation. The Administration is committed to building evidence and better integrating evidence into policy, planning, budget, operational, and management decision-making. We appreciate agencies’ efforts to strengthen the use of data and evidence to drive better decision making and achieve greater impact and efficiencies. To further these efforts, agencies should submit: 1) proposals to advance agency efforts to build and use a portfolio of evidence; and 2) proposals to strengthen agency infrastructure and capacity to use evidence, evaluation, and data as tools to improve Federal Government effectiveness. OMB M-17-28 July 7, 2017
3
Evidence-based Policy is an Overarching Objective
Evidence-Based Policymaking Commission Act of 2016 • The Promise of Evidence-Based Policymaking Final Report Released September 7, 2017
• Focus is on the use of Administrative Data – Lower costs – Reduce respondent burden – Many concepts apply to policy experiments
4
USDA Conservation Programs Seek to Improve Environmental Performance in Agriculture A portfolio of programs ….. Conservation Reserve Program (CRP) Agricultural Conservation Easement Program (ACEP)
Land Retirement/Preservation Programs • •
Environmental Quality Incentives Program (EQIP) Conservation Stewardship Program (CSP)
Working Lands Programs • •
Conservation Technical Assistance (CTA)
With multiple objectives…. Protect and improve water quality and quantity, soil health, wildlife habitat, air quality, and more
5
USDA Conservation Programs Seek to Improve Environmental Performance in Agriculture
6
All Farmers
Program Applicants
Participation Incentive What action? • Fields • Practices
All Eligible Farmers
Program Budget
Program Participants
Program Performance • Economic • Environmental • Distributional
A multitude of program design decisions affect conservation program outcomes
Eligibility • Land types • Land use • Location • Practices
What payment? • Fixed • Bid
Enrollment Screen (targeting) • Benefit-cost index • Field Location –Soils –Topography –Population • Practices • Cost (bid)
7
ERS has a long history doing, and funding, experimental work Nutrition (BEN) Agri-environmental Policy (CBEAR) Food Assistance (BECR)
Competitive Grants to Create Centers • • •
FSA CRP Nudge beginning in 2011 Behavioral Economics Master Class (2014 with Ideas 42)
Partnerships with Program Agencies • •
ERS Research and Collaboration
• Cooperative agreements (many we could list here) • ERS researcher projects and publications (e.g. 2017 Hellerstein et al ERR)
8
Economic Experiments for Policy Analysis and Program Design: A Guide for Agricultural Decisionmakers
Higgins, et al. 2017. ERS Economic Research Report 236. https://www.ers.usda.gov/publications/pub-details/?pubid=84668
9
Types of Economic Experiments
10
11
Next Steps: More Ambitious Field Experiments? Much of ERS’ field work has been on “Outreach Experiments”
• Advantages: Low cost to implement, administrative data as outcome measures, a “foot in the door” for field experiments. • Cons: Generally small treatment effects; don’t address core program design questions. • Examples: CRP Outreach, Riparian Buffers, Organic Soils (non-environ: Microloans, FSA county elections).
A more ambitious goal: Focus on program design • • • •
Examples: Auction structure, Defaults, Payment levels, Contract design. Advantages: Addresses core program design questions. Motivations: Programs are often already “trying different things.” Obstacles: More difficult to design, very difficult to get approvals, no dedicated evaluation funding (e.g. DOE).
12
• • • •
Challenges
Budget issues: Experiments are expensive and budgets are uncertain It is always a challenge, and time consuming, to work with bureaucracy (e.g., OMB implementation of the Paper Work Reduction Act) Program agencies may be constrained by mandate, regulation, capacity and culture, and concerned about privacy, equity, respondent burden, public opinion, etc. The traditional (ex-post) peer-review process is not optimized for reviewing experiments
– Assumptions underlying experimental research are imposed ex-ante, as part of the design process – Revising and repeating experiments may be prohibitively expensive
13
ERS is Adopting a Formal Process for Approving Experimental Research We designed a multi-phase process with four “gates”*
1. Specify research objective & approach, develop a timeline, and secure collaborator commitments 2. Peer review of Experimental Design and Analysis Plan -- recruitment plan, protocols & instruction, randomization protocols, modeling strategies, etc. 3. Formal administrative approvals – formal collaborator commitments, OMB review, IRB clearance 4. Implementation [5. Analysis and reporting; with the plans from above serving to discipline and justify the analytical approach]
* Following templates created by GSA’s (now defunct) Social Behavioral Science Team
14
Final Thoughts: How can (non-USDA) research make a difference?
• Know your question! • Know the decision context of individuals AND program managers • Ensure your work can withstand critical scrutiny (power tests, mechanism commitment, etc.) • Validity – field tests (including administrative experiments) are always attractive • MANY interesting questions remain unanswered… – Do defaults matters (say, when land retirement offers are solicited) – New auction mechanisms – Self monitoring & performance-based incentives
15
