Transforming Data into Insights: A Practical Guide
5/23/2017
CAUCE 2017
Transforming Data into Insights: A Practical Guide
Jean de Sousa‐Hitzler Research Analyst, Business Development and Strategic Planning
Dijana Praskac Research Analyst, Business Development and Strategic Planning
Data is interesting and exciting It allows us to understand the world around us It leads to self‐ awareness and reflection
1
5/23/2017
Data Collection and Usage Enrollment metrics Student feedback Alumni feedback Environmental scans Performance tracking Instructors feedback Labour market dynamics Employer/partner feedback
Strategic and operational planning in new program development Program reviews Program evaluations Student services Financial planning Marketing activities
Why Data? To minimize uncertainties To understand the market For accountability To stay relevant and competitive To make iterative improvements
2
5/23/2017
Evidence Based Decision‐Making A process for making decisions about a program, practice, or policy that is grounded in the best available research evidence and informed by experiential evidence from the field and relevant contextual evidence
Stages Gathering
Interpreting
Applying
Rational or Irrational?
3
5/23/2017
Students
Know Your Learners
4
5/23/2017
Learners’ Profile Typical Chang School Student: Female, 20-29, degree grad, Canadian-born, Employed FT Students are enrolling to learn new skills and to achieve personal and professional development Chang School website and word of mouth the main sources of awareness Classroom vs. Distance Ed Students DE students: Mainly female | Predominately in professional roles | Older than in-class | Canadian-born | Earning higher annual household incomes | Very likely to be university or college grads | More likely to be employed
Measuring Student Loyalty
5
5/23/2017
Millennials and the Future of Learning
6
5/23/2017
Degreed: How the Workforce Learns in 2016
Key Performance Indicators
7
5/23/2017
Performance Measurement is “the regular collection of information for monitoring how a policy, program or initiative is doing at any point in time” Treasury Board of Canada
May-25-16
Measuring impact on learners Career outcomes, satisfaction, further schooling, student loyalty
Financial/Operational excellence Internal processes and financial sustainability
Quantifying impact on community and society Outreach, city building, economic value, effect on communities, access program….
Cost‐benefit measurement Student self‐report
16
8
5/23/2017
Focusing on the metrics that matter
A Balanced Scorecard
•Revenue •Meeting operating budget •Growth rate in enrolment
• Financial • Students, Stakeholders • Programs Services and Processes • People and Culture
•Program Reviews •Website metrics •Processes mapped and redesigned
Financial
Internal Process
Goals Priorities Strategies Learner/ Stakeholder
Org. Capacity •Expenditure on staff PD •Instructor PD •Staff Innovation Events
•Satisfaction •Retention •Student Awards •New Experiential Opportunities
Designing the KPIs: Example
Performance Indicators Meeting Student Expectations Satisfaction with Experience Connection to The Chang School Continuing Education Experience Preparation for Career The Net Promoter Score
Scorecard 2016
2012
A‐ (82%)
A‐ (82%)
A (86%)
B (75%)
D (56%)
C‐ (61%)
B (80%)
B (77%)
C (64%)
C+ (67%)
+28
+26
9
5/23/2017
Challenges • Choosing the right metrics • Availability of data • Standardizing calculations/methodology • Ensuring relevance • Not easy to quantify • Clarity (requires definition) • Validation
New Program Development
10
5/23/2017
Key Research Considerations Current learners Target Audience Research
Labour market Competition
Budget
Program Development Components Institutional Capacity
Environmental Scan
Alignment with Academic Plan
Market Intelligence
Subject-matter expertise
Labour market dynamics
Resources and technology
Demand assessment – the most difficult task
Timing and financial resources
Comparative programs
Curriculum Program Advisory Committee Subject-matter expert Chang School Program Director and Academic Coordinator
Monitoring Assess outcomes (e.g., first cohort) Document processes, challenges and strengths Key Performance Indicators
11
5/23/2017
Lifecycle Analysis
Program Life‐cycle Analysis
12
5/23/2017
Methodology and Approach • Quantitative – Trend analysis of student registration, course enrolment and revenue (collaboration with Finance Group) • All certificates examined but data clustered by program area • Qualitative Analysis ‐ Trends shared with Program Directors asked to elaborate on possible reasons for observed trends (interviews and forms) • Best practices/deficiencies identified • Finance and Marketing managers consulted for insights and suggestions to derive recommendations for success
Growth
Influencing Factors •Strong targeted and collaborative marketing •Fills a distinct and current societal, professional or labour market need •Tied to professional designation •Direct relevance to career advancement or opportunities in a sector •Closely aligned to relevant target industry •Strong Academic Coordinator involvement and leadership •Format of delivery (e.g. intensives) •Unique product •Updated curriculum •Strong brand recognition
13
5/23/2017
Mature Phase Influencing Factors •Fills a distinct and current societal, professional or labour market need •Certificate directly tied to professional designation •Certificate graduates have track record of securing good employment •Strong Academic Coordinator involvement and leadership dampens any decline •Program linked to employment in multiple sectors •Convenient format of delivery (e.g. Online or Summer Intensive delivery) •Ongoing updating of curriculum ‐ even when not scheduled for periodic review •Certificate can be laddered to degree
Decline Influencing Factors •Review process sporadic or overdue •Certificate not tied to professional advancement or employment opportunities •Inconsistent or sporadic course scheduling •Inadequate marketing strategy •Weak support from home department or academic coordinator •Evolving sector and labour market needs •External factors such as increased competition, market saturation or economic fluctuations
14
5/23/2017
How can we use this? Area
I
G
M
D
1
29%
7%
36%
29%
2
48%
14%
28%
14%
3
7%
28%
49%
14%
4
25%
33%
33%
8%
5
36%
27%
27%
9%
Phases
Implications for Management
Introductory
•Investment in targeted marketing and promotion •Coordinated and targeted launch (maximise publicity) •Reach out to industry and community stakeholders
Growth
•Investment in quality and student satisfaction •Maintain competitive intelligence (monitor mar/competitors)
Mature
•Investment in reviewing, optimizing and rejuvenating of certificate •Engage in stakeholder consultations •Focus on differentiation to maintain market share •Diversify modes of delivery •Engage in more frequent periodic reviews to promote rejuvenation of certificates
Decline
•Investment in comprehensive reviewing and rejuvenating of certificate •Recommend obsolescence* •Renewal of program pipeline
Factors for Success?
15
5/23/2017
16
CAUCE 2017
Transforming Data into Insights: A Practical Guide
Jean de Sousa‐Hitzler Research Analyst, Business Development and Strategic Planning
Dijana Praskac Research Analyst, Business Development and Strategic Planning
Data is interesting and exciting It allows us to understand the world around us It leads to self‐ awareness and reflection
1
5/23/2017
Data Collection and Usage Enrollment metrics Student feedback Alumni feedback Environmental scans Performance tracking Instructors feedback Labour market dynamics Employer/partner feedback
Strategic and operational planning in new program development Program reviews Program evaluations Student services Financial planning Marketing activities
Why Data? To minimize uncertainties To understand the market For accountability To stay relevant and competitive To make iterative improvements
2
5/23/2017
Evidence Based Decision‐Making A process for making decisions about a program, practice, or policy that is grounded in the best available research evidence and informed by experiential evidence from the field and relevant contextual evidence
Stages Gathering
Interpreting
Applying
Rational or Irrational?
3
5/23/2017
Students
Know Your Learners
4
5/23/2017
Learners’ Profile Typical Chang School Student: Female, 20-29, degree grad, Canadian-born, Employed FT Students are enrolling to learn new skills and to achieve personal and professional development Chang School website and word of mouth the main sources of awareness Classroom vs. Distance Ed Students DE students: Mainly female | Predominately in professional roles | Older than in-class | Canadian-born | Earning higher annual household incomes | Very likely to be university or college grads | More likely to be employed
Measuring Student Loyalty
5
5/23/2017
Millennials and the Future of Learning
6
5/23/2017
Degreed: How the Workforce Learns in 2016
Key Performance Indicators
7
5/23/2017
Performance Measurement is “the regular collection of information for monitoring how a policy, program or initiative is doing at any point in time” Treasury Board of Canada
May-25-16
Measuring impact on learners Career outcomes, satisfaction, further schooling, student loyalty
Financial/Operational excellence Internal processes and financial sustainability
Quantifying impact on community and society Outreach, city building, economic value, effect on communities, access program….
Cost‐benefit measurement Student self‐report
16
8
5/23/2017
Focusing on the metrics that matter
A Balanced Scorecard
•Revenue •Meeting operating budget •Growth rate in enrolment
• Financial • Students, Stakeholders • Programs Services and Processes • People and Culture
•Program Reviews •Website metrics •Processes mapped and redesigned
Financial
Internal Process
Goals Priorities Strategies Learner/ Stakeholder
Org. Capacity •Expenditure on staff PD •Instructor PD •Staff Innovation Events
•Satisfaction •Retention •Student Awards •New Experiential Opportunities
Designing the KPIs: Example
Performance Indicators Meeting Student Expectations Satisfaction with Experience Connection to The Chang School Continuing Education Experience Preparation for Career The Net Promoter Score
Scorecard 2016
2012
A‐ (82%)
A‐ (82%)
A (86%)
B (75%)
D (56%)
C‐ (61%)
B (80%)
B (77%)
C (64%)
C+ (67%)
+28
+26
9
5/23/2017
Challenges • Choosing the right metrics • Availability of data • Standardizing calculations/methodology • Ensuring relevance • Not easy to quantify • Clarity (requires definition) • Validation
New Program Development
10
5/23/2017
Key Research Considerations Current learners Target Audience Research
Labour market Competition
Budget
Program Development Components Institutional Capacity
Environmental Scan
Alignment with Academic Plan
Market Intelligence
Subject-matter expertise
Labour market dynamics
Resources and technology
Demand assessment – the most difficult task
Timing and financial resources
Comparative programs
Curriculum Program Advisory Committee Subject-matter expert Chang School Program Director and Academic Coordinator
Monitoring Assess outcomes (e.g., first cohort) Document processes, challenges and strengths Key Performance Indicators
11
5/23/2017
Lifecycle Analysis
Program Life‐cycle Analysis
12
5/23/2017
Methodology and Approach • Quantitative – Trend analysis of student registration, course enrolment and revenue (collaboration with Finance Group) • All certificates examined but data clustered by program area • Qualitative Analysis ‐ Trends shared with Program Directors asked to elaborate on possible reasons for observed trends (interviews and forms) • Best practices/deficiencies identified • Finance and Marketing managers consulted for insights and suggestions to derive recommendations for success
Growth
Influencing Factors •Strong targeted and collaborative marketing •Fills a distinct and current societal, professional or labour market need •Tied to professional designation •Direct relevance to career advancement or opportunities in a sector •Closely aligned to relevant target industry •Strong Academic Coordinator involvement and leadership •Format of delivery (e.g. intensives) •Unique product •Updated curriculum •Strong brand recognition
13
5/23/2017
Mature Phase Influencing Factors •Fills a distinct and current societal, professional or labour market need •Certificate directly tied to professional designation •Certificate graduates have track record of securing good employment •Strong Academic Coordinator involvement and leadership dampens any decline •Program linked to employment in multiple sectors •Convenient format of delivery (e.g. Online or Summer Intensive delivery) •Ongoing updating of curriculum ‐ even when not scheduled for periodic review •Certificate can be laddered to degree
Decline Influencing Factors •Review process sporadic or overdue •Certificate not tied to professional advancement or employment opportunities •Inconsistent or sporadic course scheduling •Inadequate marketing strategy •Weak support from home department or academic coordinator •Evolving sector and labour market needs •External factors such as increased competition, market saturation or economic fluctuations
14
5/23/2017
How can we use this? Area
I
G
M
D
1
29%
7%
36%
29%
2
48%
14%
28%
14%
3
7%
28%
49%
14%
4
25%
33%
33%
8%
5
36%
27%
27%
9%
Phases
Implications for Management
Introductory
•Investment in targeted marketing and promotion •Coordinated and targeted launch (maximise publicity) •Reach out to industry and community stakeholders
Growth
•Investment in quality and student satisfaction •Maintain competitive intelligence (monitor mar/competitors)
Mature
•Investment in reviewing, optimizing and rejuvenating of certificate •Engage in stakeholder consultations •Focus on differentiation to maintain market share •Diversify modes of delivery •Engage in more frequent periodic reviews to promote rejuvenation of certificates
Decline
•Investment in comprehensive reviewing and rejuvenating of certificate •Recommend obsolescence* •Renewal of program pipeline
Factors for Success?
15
5/23/2017
16
