Surveys, Surveys
Surveys, Surveys MAXIMIZING THE POWER OF PERCEPTUAL DATA
#AQuESTT
Goal 2 SLDS Grant Provide a statewide system of professional development training for data analysis that reaches every district.
Tiered Training Delivery Statewide Data Cadre ESUs and NDE Staff School District Leadership and Staff
Statewide Data Cadre ESUs/ESUCC •
Rhonda Jindra - ESU 1
•
Mike Danahy - ESU 2
•
Marilou Jasnoch - ESU 3
•
Annette Weise - ESU 5
•
Lenny VerMaas - ESU 6
•
Denise O’Brien - ESU 10
•
Melissa Engel - ESU 16
•
Jeff McQuistan - ESU 17
NDE Data, Research, Evaluation (DRE) • • • •
Kunal Dash Matt Heusman Russ Masco Justine Yeo
Network Education and Technology (NEaT) • Kristina Peters Federal Programs • Beth Zillig Special Education • Teresa Coontz
Higher Education •
Dick Meyer - UNK
NDE Accreditation and School Improvement • Sue Anderson • Don Loseke Assessment • John Moon Curriculum • Cory Epler • Tricia Parker-Siemers
Objectives 1. 2. 3. 4. 5.
Understand the role that perceptual data plays. Establish a process for perceptual data collection Learn how to analyze text data generated from open-ended questions Share existing data collection resources Learn how to increase your response rate
1. The Role of Perceptual Data Why Survey? It is REQUIRED! • AdvancED requires the administration of ASSIST surveys within 1 year of the External Review • Nebraska Frameworks should utilize perceptual data to be a part of the district profile for the external visit
It is one of the four lenses Multiple Measures of Data • • • •
Demographics School Processes Student Learning Perceptions
It answers the question
“How do we do business?” Where we are now? What do students, staff, and parents think about the learning environment?
Perceptions impact beliefs, behaviors, and, to an extent, outcomes.
2. Collection Process 1. 2. 3. 4.
Identify the Purpose Determine the Content Develop or Select the Instrument Pilot the Instrument
Identify the Purpose What do you want to learn? 2. Will it be important to monitor changes in perceptions over time? How often? For how long? 3. How will you use your survey results? 1.
Determine the Content • What do you need to know? • What do you want to know? • Which stakeholders’ perspectives are relevant and should be considered?
Select/Develop and Pilot the Instrument 1. Who is the pilot group? 2. Is the questionnaire working the way you want it to work?
Collect the Data ●
●
How will the survey be administered? (paper/pencil, web-based) When will the survey be administered?
How do we organize and display the data? • How can the data be graphed and reported to objectively and fairly demonstrate the results?
• The goal of data display is to inform the audience and create consistent understanding.
Displaying the Results
How do we interpret the data? 1. How can the results be analyzed to show the information gleaned from the survey? 2. What did we learn about our learning environment? 3. What connections can be made when correlated with other data sources? (demographic, school processes, student learning)
How do we report the results? • How are the results going to be shared? • When are the results going to be shared? • When/How do you report changes that were made due to these results?
How are you going to use the results? • How do these results help the CIP process? • What can you learn from these results? • How do we change perceptions?
3. Analyzing Text Data Use Text Mining (text analytics) ●
●
●
Determining the frequency of positive and negative sounding words Making qualitative (unstructured) data usable by a computer “helps derive potentially valuable insights from text-based content”
Source: http://searchbusinessanalytics.techtarget.com/definition/text-mining
Analyzing Text Data Did you know? ● ●
●
An estimated 80% of data is ‘unstructured’? Sources include Emails, articles, internal reports, transcripts of phone calls, research papers, blog entries, patent applications. More than 7 million web pages of text are added to a collective repository daily.
Analyzing Text Data ● ●
YouTube Tutorial Examples of open source software that can be used for analyzing text data ○ ○ ○ ○
GATE KH Coder OpenNLP KNIME Text Processing
Source: http://www.predictiveanalyticstoday.com/top-11-free-software-for-text-analysistextmining-text-analytics/
Analyzing Text Data
●
Refer to handout
4. Available Resources What instruments have already been developed? 1. ASSIST Surveys (AdvancED) 2. State Surveys (NDE)
NDE Perceptual Surveys
Displaying Perceptual Data Results Page 248
NDE Perceptual Surveys Response numbers from a few perceptual surveys administered (as of 4/8/2015)
NDE Perceptual Surveys • Ability to administer surveys will be available in future years as well • NDE’s capacity to manage the perceptual data surveys is developing • Development of “mini-surveys” by constructs (main ideas being measured)
5. Survey Response Rate 1. 2. 3. 4. 5. 6.
Causes for Low Response Strategies to Increase Survey Response Survey Constructs Web-based Survey Guidelines NDE Assistance Sources of Survey Error
Sources of Survey Error What are some potential sources of survey error?
Source: https://www.qualtrics.com/blog/sources-of-error-in-survey-research/
Sources of Survey Error
Source: http://www.tpsgc-pwgsc.gc.ca/rop-por/rapports-reports/telephone/introduction-eng.html
Causes for High Non-response ● ● ● ● ● ● ●
Not Targeting the Desired Population No Pre-defined Respondents Lack of Survey Topic Saliency Improper Visual Design No Prior Communication or Notification Survey Interpretation Web-based: Internet Speed, Browser Compatibility, Screen Resolution
Increasing Survey Response ● ● ●
● ● ●
Targeting the Correct Population Identifying a Specific Sampling Frame Establishing Relevance to Targeted Population Incentivizing the Respondents Pre-notifying Targeted Population Building your own web-based: Internet speed, browser compatibility and consistency
Survey Constructs What is being measured? Some constructs that are measured by perceptual surveys include the following: For Students
For Staff
For Parents
Safety
School Environment
Children’s Safety
Freedom of Choice
Administrative Support
School Staff
Relationship with Teachers
Staff Morale
School Assessment
Relationship with Other Students
Relationship with Students
Learning Environment
School Performance
Student Achievement
Family Support
Web-based Survey Guidelines When developing a survey it is important to decide exactly what you want to measure and design the survey accordingly. Are you measuring . . . ● Behavior ● Attitude ● Knowledge ● Demographic Information
Web-based Survey Guidelines • Define terms • Avoid question ambiguity and ‘leading’ questions • Provide response options at end of questions • Provide clear question objectives • Reasonable question length
• Avoid asking two questions in one • Use simple vocabulary • Avoid calculations • Reference periods should be clear and narrow • Use specific, rather than general, questions • Use an odd-numbered scale
NDE Assistance The NDE Data, Research, and Evaluation (DRE) Team can assist you: ● ● ● ●
Creating New Surveys Proofing and reviewing survey instruments Survey data analyses Survey construct analysis
● ● ● ●
Survey management process assistance Addressing sources of survey error Survey software recommendations Survey question and design development
Thank you! #AQuESTT
#AQuESTT
Goal 2 SLDS Grant Provide a statewide system of professional development training for data analysis that reaches every district.
Tiered Training Delivery Statewide Data Cadre ESUs and NDE Staff School District Leadership and Staff
Statewide Data Cadre ESUs/ESUCC •
Rhonda Jindra - ESU 1
•
Mike Danahy - ESU 2
•
Marilou Jasnoch - ESU 3
•
Annette Weise - ESU 5
•
Lenny VerMaas - ESU 6
•
Denise O’Brien - ESU 10
•
Melissa Engel - ESU 16
•
Jeff McQuistan - ESU 17
NDE Data, Research, Evaluation (DRE) • • • •
Kunal Dash Matt Heusman Russ Masco Justine Yeo
Network Education and Technology (NEaT) • Kristina Peters Federal Programs • Beth Zillig Special Education • Teresa Coontz
Higher Education •
Dick Meyer - UNK
NDE Accreditation and School Improvement • Sue Anderson • Don Loseke Assessment • John Moon Curriculum • Cory Epler • Tricia Parker-Siemers
Objectives 1. 2. 3. 4. 5.
Understand the role that perceptual data plays. Establish a process for perceptual data collection Learn how to analyze text data generated from open-ended questions Share existing data collection resources Learn how to increase your response rate
1. The Role of Perceptual Data Why Survey? It is REQUIRED! • AdvancED requires the administration of ASSIST surveys within 1 year of the External Review • Nebraska Frameworks should utilize perceptual data to be a part of the district profile for the external visit
It is one of the four lenses Multiple Measures of Data • • • •
Demographics School Processes Student Learning Perceptions
It answers the question
“How do we do business?” Where we are now? What do students, staff, and parents think about the learning environment?
Perceptions impact beliefs, behaviors, and, to an extent, outcomes.
2. Collection Process 1. 2. 3. 4.
Identify the Purpose Determine the Content Develop or Select the Instrument Pilot the Instrument
Identify the Purpose What do you want to learn? 2. Will it be important to monitor changes in perceptions over time? How often? For how long? 3. How will you use your survey results? 1.
Determine the Content • What do you need to know? • What do you want to know? • Which stakeholders’ perspectives are relevant and should be considered?
Select/Develop and Pilot the Instrument 1. Who is the pilot group? 2. Is the questionnaire working the way you want it to work?
Collect the Data ●
●
How will the survey be administered? (paper/pencil, web-based) When will the survey be administered?
How do we organize and display the data? • How can the data be graphed and reported to objectively and fairly demonstrate the results?
• The goal of data display is to inform the audience and create consistent understanding.
Displaying the Results
How do we interpret the data? 1. How can the results be analyzed to show the information gleaned from the survey? 2. What did we learn about our learning environment? 3. What connections can be made when correlated with other data sources? (demographic, school processes, student learning)
How do we report the results? • How are the results going to be shared? • When are the results going to be shared? • When/How do you report changes that were made due to these results?
How are you going to use the results? • How do these results help the CIP process? • What can you learn from these results? • How do we change perceptions?
3. Analyzing Text Data Use Text Mining (text analytics) ●
●
●
Determining the frequency of positive and negative sounding words Making qualitative (unstructured) data usable by a computer “helps derive potentially valuable insights from text-based content”
Source: http://searchbusinessanalytics.techtarget.com/definition/text-mining
Analyzing Text Data Did you know? ● ●
●
An estimated 80% of data is ‘unstructured’? Sources include Emails, articles, internal reports, transcripts of phone calls, research papers, blog entries, patent applications. More than 7 million web pages of text are added to a collective repository daily.
Analyzing Text Data ● ●
YouTube Tutorial Examples of open source software that can be used for analyzing text data ○ ○ ○ ○
GATE KH Coder OpenNLP KNIME Text Processing
Source: http://www.predictiveanalyticstoday.com/top-11-free-software-for-text-analysistextmining-text-analytics/
Analyzing Text Data
●
Refer to handout
4. Available Resources What instruments have already been developed? 1. ASSIST Surveys (AdvancED) 2. State Surveys (NDE)
NDE Perceptual Surveys
Displaying Perceptual Data Results Page 248
NDE Perceptual Surveys Response numbers from a few perceptual surveys administered (as of 4/8/2015)
NDE Perceptual Surveys • Ability to administer surveys will be available in future years as well • NDE’s capacity to manage the perceptual data surveys is developing • Development of “mini-surveys” by constructs (main ideas being measured)
5. Survey Response Rate 1. 2. 3. 4. 5. 6.
Causes for Low Response Strategies to Increase Survey Response Survey Constructs Web-based Survey Guidelines NDE Assistance Sources of Survey Error
Sources of Survey Error What are some potential sources of survey error?
Source: https://www.qualtrics.com/blog/sources-of-error-in-survey-research/
Sources of Survey Error
Source: http://www.tpsgc-pwgsc.gc.ca/rop-por/rapports-reports/telephone/introduction-eng.html
Causes for High Non-response ● ● ● ● ● ● ●
Not Targeting the Desired Population No Pre-defined Respondents Lack of Survey Topic Saliency Improper Visual Design No Prior Communication or Notification Survey Interpretation Web-based: Internet Speed, Browser Compatibility, Screen Resolution
Increasing Survey Response ● ● ●
● ● ●
Targeting the Correct Population Identifying a Specific Sampling Frame Establishing Relevance to Targeted Population Incentivizing the Respondents Pre-notifying Targeted Population Building your own web-based: Internet speed, browser compatibility and consistency
Survey Constructs What is being measured? Some constructs that are measured by perceptual surveys include the following: For Students
For Staff
For Parents
Safety
School Environment
Children’s Safety
Freedom of Choice
Administrative Support
School Staff
Relationship with Teachers
Staff Morale
School Assessment
Relationship with Other Students
Relationship with Students
Learning Environment
School Performance
Student Achievement
Family Support
Web-based Survey Guidelines When developing a survey it is important to decide exactly what you want to measure and design the survey accordingly. Are you measuring . . . ● Behavior ● Attitude ● Knowledge ● Demographic Information
Web-based Survey Guidelines • Define terms • Avoid question ambiguity and ‘leading’ questions • Provide response options at end of questions • Provide clear question objectives • Reasonable question length
• Avoid asking two questions in one • Use simple vocabulary • Avoid calculations • Reference periods should be clear and narrow • Use specific, rather than general, questions • Use an odd-numbered scale
NDE Assistance The NDE Data, Research, and Evaluation (DRE) Team can assist you: ● ● ● ●
Creating New Surveys Proofing and reviewing survey instruments Survey data analyses Survey construct analysis
● ● ● ●
Survey management process assistance Addressing sources of survey error Survey software recommendations Survey question and design development
Thank you! #AQuESTT
