APPLIED LEARNING STUDENT QUESTIONNAIRE: OVERALL ANALYSIS Overall Results May 2013 Executive Summary Participants and Methods In May 2013, 928 students across 6 Race to the Top programs completed the Applied Learning Student Questionnaire (ALSQ). The response rates displayed in Table 1 suggest that 90% of the total number of participating students were successfully surveyed. Table 1. Survey Response Rates Program STEM for Life Carroll County Drew Charter School- Partners of Innovation Murray County STEM Academy 21st Century STEM Collaboration- Barrow County STEM Targeted Education Program (STEP) Academy- Sweetwater MS and Moore MS Tift County Mechatronics Program Total # of Survey Respondents 160 273 37 318 Total # of Participating Students 160 301 50 365 Survey Response Rate 100% 91% 74% 87% 109 121 90% 31 36 86% 928 1033 90% The ALSQ1 is designed to measure pre and post gains related to student problem solving and communication skills, self-management and engagement. The ALSQ is a self-report questionnaire that includes 36 items to assess students' attitudes on the following survey constructs: 1. Intrinsic Motivation: motivation stemming from goals of mastery, learning and challenge. Example, "It is important for me to learn what is being taught in this program." 2. Self-management/Self-Regulation: effortful and persistent behaviors that are used to guide, monitor, and direct the success of one's learning and performance. Example, "I turn all my assignments in on time." 3. Intent to Persist: aspirations, plans, and goals to pursue additional education and a career in STEM. Example, "I intend to get a college degree in STEM (Science, Technology, Engineering, and Math)." 4. Problem Solving: inquiry-based learning environment that provides higher-order cognitive tasks and real-world applications. Example, "I work out explanations on my own." 5. Implementation Activities: hands-on activities designed to increase exposure to STEM topics and real-world applications. Example, "We learn what scientists/technicians/engineers/ mathematicians or other STEM professionals do." 1 See Appendix A for information related to the construct reliabilities of the ALSQ. Prepared by The Findings Group, LLC Page 1 Executive Summary, continued Results & Discussion ALSQ Survey Constructs Table 2 summarizes students' responses to the ALSQ survey constructs across all programs. It is clear that the programs were effective at producing statistically significant increases in students' intrinsic motivation, self-management/self-regulation skills and intent to persist. The largest student gains observed were in the intrinsic motivation construct. Before the program, less than 53% of students indicated that they derive value and see the importance in learning about STEM; now, more than 76% say that they are intrinsically motivated to tackle STEM-related tasks and projects. Despite these statistically significant gains, it is important to note that the "now" scores across the following 3 constructs did not reach or exceed the optimal average of 4.0 on a 5-point Likert scale (1, strongly disagree to 5, strongly agree): Intent to Persist, Problem Solving, and Implementation Activities. See Figure 1. In order to maximize effectiveness, we would expect students' average scores to exceed 4.0. Figure 1 suggests that additional work may be needed in the above mentioned areas. Table 2. Summary of Results by Constructs Constructs n Before 920 Intrinsic Motivation Now 912 Overall- Constructs Mean1 Paired Samples t-test 1 (Strongly Disagree) 2 (Disagree) 3 (Neutral) 4 (Agree) 5 (Strongly Agree) 3.54 0.000** 4.10 +N+N+N+N+N+ +N+N+N+N+N+ 5% 2% 10% 32% 31% 22% 4% 17% 34% 42% Self- Management/Self- Regulation Intent to Persist Before 919 Now 906 Before 915 Now 906 3.83 0.000** 4.11 +N+N+N+N+N+ +N+N+N+N+N+ 19% 20% 12% 9% 25% 26% 18% 27% 18% 27% 3.38 0.000** 3.69 +N+N+N+N+N+ +N+N+N+N+N+ 12% 10% 14% 10% 26% 17% 22% 19% 30% 39% Problem Solving Now 913 3.93 N/A +N+N+N+N+N+ 3% 5% 22% 36% 34% Implementation activities Now 911 3.84 N/A +N+N+N+N+N+ 3% 6% 24% 36% 30% 1 Reference lines are set at 3.5 and 4. Negatively worded statements were reverse coded for mean computations. **p<.01, *p<.05, p<.10 5.00 Figure 1. Constructs Before Now Optimal Average 4.50 ** 4.00 ** ** 3.50 3.00 2.50 2.00 1, Strongly Disagree to 5, Strongly Agree Intrinsic Motivation Self- Management/Self- Regulation Intent to Persist Problem Solving Implementation Activitites **p<.01, *p<.05, p<.10; Scale is truncated for visual clarity. Prepared by The Findings Group, LLC Constructs Page 2 Executive Summary, continued ALSQ Survey Constructs by Program Examining the ALSQ results by individual program, it is evident that all programs, with the exception of Murray STEM Academy, show statistically significant increases in intrinsic motivation, self- management/self-regulation and intent to persist. Students in the STEM for Life program at Carroll County and the Mechatronics program at Tift County show the largest increases from before to now on all three of the above mentioned constructs; Murray STEM Academy students show the smallest average increases across all 6 programs. See Table 3. Table 3. Summary of Results by Constructs per Program Overall- Constructs per Program Constructs STEM for Life Carroll County (n=160) Drew Charter (n=273) Murray STEM Academy (n=37) 21st Century Barrow County (n=318) STEP Academy Moore MS Sweetwater MS (n=109) TIFT County Mechatronics (n=31) Mean t-test Mean t-test Mean t-test Mean t-test Mean t-test Mean t-test Intrinsic Motivation Before Now 3.42 4.16 0.000** 3.75 4.16 0.000** 3.04 3.05 0.933 3.46 4.09 0.000** 3.41 4.12 0.000** 4.06 4.75 0.000** Self- Before 3.72 3.94 3.47 3.89 3.61 4.08 Management/ Self-Regulation Now 4.22 0.000** 4.15 0.000** 3.55 0.512 4.09 0.000** 4.02 0.000** 4.50 0.000** Intent to Persist Before Now 3.22 3.64 0.000** 3.63 3.82 0.000** 2.83 2.84 0.908 3.29 3.62 0.000** 3.36 3.68 0.000** 3.79 4.59 0.000** Problem Solving Now 3.92 3.99 3.00 3.99 3.77 4.60 Implementation activities Now 3.81 n/a 3.90 n/a 2.91 n/a 3.93 n/a 3.72 n/a 4.57 n/a Scale= 1, Strongly Disagree to 5, Strongly Agree. Negatively worded statements were reverse coded for mean computations. **p<.01, *p<.05, p<.10 In order for programs to maximize their effectiveness, we would expect "now" scores to reach or exceed the optimal average of 4.0. Figures 2 6 display "now" scores for each program and construct. For example, Figure 2 indicates that all programs met or exceeded the optimal average for intrinsic motivation, with the exception of Murray STEM Academy. In general, programs not reaching or exceeding the red horizontal line may need additional attention. Figure 2. Intrinsic Motivation ("Now" Scores) 5 Figure 3. Self-Management/Self-Regulation ("Now" Scores) 5 4.5 4.5 4 4 3.5 3.5 3 3 2.5 2.5 Scale= 1, Strongly Disagree to 5, Strongly Agree. Scale was truncated for visual clarity. STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore and Sweetwater MS TIFT County Mechatronics STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore and Sweetwater MS TIFT County Mechatronics Prepared by The Findings Group, LLC Page 3 Executive Summary, continued Figure 4. Intent to Persist ("Now" Scores) 5 4.5 4 3.5 3 2.5 Figure 5. Problem Solving ("Now" Scores) 5 4.5 4 3.5 3 2.5 STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore and Sweetwater MS TIFT County Mechatronics STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore and Sweetwater MS TIFT County Mechatronics STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore and Sweetwater MS TIFT County Mechatronics STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore and Sweetwater MS TIFT County Mechatronics Figure 6. Implementation Activities ("Now" Scores 5 Figure 7. Overall Ratings 5 4.5 4.5 4 4 3.5 3.5 3 3 2.5 2.5 2 Scale= 1, Strongly Disagree to 5, Strongly Agree. Scale was truncated for visual clarity. Program Rating Scale= 1, Very Poor to 5, Excellent. Program Rating Collapsing across all programs, students' ratings exceeded the optimal average of 4.0. On a 5-point Likert scale where 1 signifies very poor and 5 signifies excellent, the average score was a 4.20. Looking at Figure 7, above, we see that 5 out of 6 programs were rated highly. However, Murray County STEM Academy may need additional assistance in improving student enjoyment. Areas for Further Improvement Across all programs, further enhancing problem solving skills may be warranted. Specifically, students' ratings suggest that the inquiry-based learning environment may be improved by allowing students more opportunity to choose their own topics, work out explanations on their own, and plan and conduct their own projects. Likewise, encouraging programs to provide activities that foster interaction with STEM professionals may increase student exposure to real-world applications and careers. Such implementation activities may strengthen students' intentions and motivations to pursue additional education in STEM fields. Prepared by The Findings Group, LLC Page 4 Table 4. Intrinsic Motivation Intrinsic Motivation n 1. I prefer class work that is challenging so I can learn new things. Before 920 Now 909 2. It is important to me to learn what is being taught in this program. Before 919 Now 906 3. I like what I am learning in this program. Before 913 Now 904 4. I think I will be able to use what I learn in this program in other classes. Before 913 Now 897 5. Even when I do poorly on a test, Before 920 I try to learn from my mistakes. Now 912 6. I think that what I am learning in Before 906 this program is useful for me to know. Now 898 7. I think that what we are learning Before 911 in this program is interesting. Now 907 8. Understanding STEM (Science, Technology, Engineering, and Math) is important to me. Before 918 Now 907 9. I enjoy STEM (Science, Before 915 Technology, Engineering, and Math) in general. Now 906 1 Reference lines are set at 3.5 and 4. **p<.01, *p<.05, p<.10 Mean1 Paired Samples t-test 1 (Strongly Disagree) 2 3 (Disagree) (Neutral) 4 (Agree) 5 (Strongly Agree) 3.05 +N+N+N+N+N+ 11% 3.74 0.000** +N+N+N+N+N+ 4% 17% 39% 23% 10% 6% 26% 40% 24% 3.83 +N+N+N+N+N+ 4% 4.33 0.000** +N+N+N+N+N+ 1% 5% 24% 39% 28% 2% 12% 34% 52% 3.55 4.12 0.000** +N+N+N+N+N+ +N+N+N+N+N+ 4% 2% 3.59 +N+N+N+N+N+ 4% 4.19 0.000** +N+N+N+N+N+ 1% 9% 35% 33% 19% 3% 18% 35% 42% 10% 30% 36% 20% 4% 14% 36% 45% 3.94 4.40 0.000** +N+N+N+N+N+ +N+N+N+N+N+ 3% 1% 3.63 +N+N+N+N+N+ 4% 4.23 0.000** +N+N+N+N+N+ 2% 6% 21% 35% 35% 2% 10% 30% 57% 9% 31% 33% 24% 3% 13% 33% 48% 3.41 3.97 0.000** +N+N+N+N+N+ +N+N+N+N+N+ 7% 3% 3.53 +N+N+N+N+N+ 6% 4.10 0.000** +N+N+N+N+N+ 2% 11% 35% 31% 17% 5% 19% 38% 36% 10% 34% 25% 25% 4% 20% 32% 43% 3.31 +N+N+N+N+N+ 8% 3.87 0.000** +N+N+N+N+N+ 4% 13% 36% 27% 16% 6% 23% 32% 34% Prepared by The Findings Group, LLC Page 5 Table 5. Self-Regulation/Self-Motivation Self-Regulation/Self-Motivation n Mean1 Paired Samples t-test 1 (Strongly Disagree) 2 3 (Disagree) (Neutral) 4 (Agree) 5 (Strongly Agree) 10. I turn all my assignments in on Before 919 time. Now 904 3.48 3.87 0.000** +N+N+N+N+N+ +N+N+N+N+N+ 3% 2% 12% 34% 33% 17% 7% 23% 40% 29% 11. I miss class often. (n) Before 911 Now 898 1.75 1.61 0.000** +N+N+N+N+N+ +N+N+N+N+N+ 60% 66% 19% 11% 7% 3% 17% 10% 4% 3% 12. I am often late for class. (n) Before 900 Now 891 1.74 1.62 0.000** +N+N+N+N+N+ +N+N+N+N+N+ 57% 62% 22% 13% 5% 3% 21% 11% 4% 2% 13. I set aside time to do my homework and study. Before 917 Now 903 3.30 3.66 0.000** +N+N+N+N+N+ +N+N+N+N+N+ 8% 6% 13% 34% 32% 13% 7% 26% 35% 26% 14. When I say I'm going to do something, I do it. Before 914 Now 906 3.81 4.14 0.000** +N+N+N+N+N+ +N+N+N+N+N+ 2% 1% 5% 30% 34% 29% 3% 20% 34% 42% 15. I am a hard worker. Before 914 Now 902 3.96 4.29 0.000** +N+N+N+N+N+ +N+N+N+N+N+ 2% 1% 5% 22% 37% 34% 3% 12% 34% 50% 16. I finish whatever I begin. Before 911 Now 904 3.73 4.05 0.000** +N+N+N+N+N+ +N+N+N+N+N+ 2% 2% 1 Reference lines are set at 3.5 and 4. **p<.01, *p<.05, p<.10; (n) nega vely worded statement 8% 31% 33% 26% 3% 22% 36% 38% Prepared by The Findings Group, LLC Page 6 Table 6. Intent to Persist Intent to Persist n 17. I am considering a career in STEM (Science, Technology, Engineering, and Math). Before 914 Now 905 18. I intend to get a college degree Before 915 in STEM (Science, Technology, Engineering, and Math). Now 904 19. I can see myself working in STEM (Science, Technology, Engineering, and Math). Before 913 Now 901 20. Someday, I would like to have a Before 913 career in STEM (Science, Technology, Engineering, and Math). Now 900 21. I intend to graduate from high Before 912 school Now 906 1 Reference lines are set at 3.5 and 4. **p<.01, *p<.05, p<.10 Mean1 Paired Samples t-test 1 (Strongly Disagree) 2 3 (Disagree) (Neutral) 4 (Agree) 5 (Strongly Agree) 3.03 0.000** +N+N+N+N+N+ 16% 18% 30% 18% 18% 3.39 +N+N+N+N+N+ 14% 11% 25% 21% 28% 3.15 0.000** +N+N+N+N+N+ 13% 3.49 +N+N+N+N+N+ 10% 16% 33% 20% 18% 10% 29% 22% 29% 3.03 0.000** +N+N+N+N+N+ 15% 3.39 +N+N+N+N+N+ 12% 17% 31% 21% 15% 12% 26% 27% 24% 2.99 +N+N+N+N+N+ 16% 0.000** 3.36 +N+N+N+N+N+ 11% 18% 33% 18% 15% 14% 27% 22% 25% 4.72 0.000** +N+N+N+N+N+ 1% 4.83 +N+N+N+N+N+ 1% 2% 4% 9% 84% 1% 2% 5% 91% Prepared by The Findings Group, LLC Page 7 Table 7. Problem Solving, Now Only 1 Problem Solving n Mean Assessment (Strongly Disagree) 22. In this program, my teacher(s) tells me how to improve my 913 4.17 Good +N+N+N+N+N+ 2% work. 23. In this program, my teacher(s) lets us choose our own topics 908 3.39 Action ! +N+N+N+N+N+ 7% or projects to investigate. 24. In this program, I work out explanations on my own. 911 3.67 Attention +N+N+N+N+N+ 2% 25. In this program, I have opportunities to explain my 913 3.93 Attention +N+N+N+N+N+ 2% ideas. 26. In this program, we plan and do our own projects and/or 909 3.61 Attention +N+N+N+N+N+ 4% experiments. 27. In this program, we work on real-world problems. 911 3.90 Attention +N+N+N+N+N+ 3% 28. In this program, we have class discussions. 913 4.19 Good +N+N+N+N+N+ 3% 29. In this program, we investigate to see if our ideas are right. 911 4.00 Good +N+N+N+N+N+ 2% 30. In this program, we need to be able to think and ask questions. 909 4.28 Good +N+N+N+N+N+ 2% 31. In this program, we are expected to understand and 913 4.17 Good +N+N+N+N+N+ 2% explain ideas. 1 Reference lines are set at 3.5 and 4. Assessment: Good=Above 4.0; Attention=Below 4.0; Action=Below 3.5. 2 (Disagree) 2% 13% 5% 5% 11% 6% 3% 5% 2% 2% 3 (Neutral) 15% 33% 35% 23% 31% 23% 14% 20% 13% 18% 4 (Agree) 38% 27% 41% 40% 30% 36% 33% 39% 37% 36% 5 (Strongly Agree) 42% 19% 17% 30% 25% 33% 47% 35% 48% 43% Prepared by The Findings Group, LLC Page 8 Table 8. Implementation Activities, Now Only 1 Implementation Activities n Mean Assessment (Strongly Disagree) 32. In this program, my teacher(s) takes notice of 911 3.89 Attention +N+N+N+N+N+ 4% students' ideas. 33. In this program, my teacher(s) shows us how new information relates to what 905 4.15 Good +N+N+N+N+N+ 2% we have already learned. 34. In this program, we learn what scientists/ technicians/ engineers/ mathematicians 909 3.77 Attention +N+N+N+N+N+ 4% or other STEM professionals do. 35. In this program, we do our work in groups. 907 3.71 Attention +N+N+N+N+N+ 3% 36. In this program, we interact with scientists/ technicians/ engineers/ mathematicians 908 3.70 Attention +N+N+N+N+N+ 5% or other STEM professionals. 1 Reference lines are set at 3.5 and 4. Assessment: Good=Above 4.0; Attention=Below 4.0; Action=Below 3.5. 2 (Disagree) 5% 3% 7% 6% 9% 3 (Neutral) 22% 16% 25% 33% 25% 4 (Agree) 38% 39% 36% 36% 32% 5 (Strongly Agree) 32% 41% 28% 23% 29% Table 9. Educational Plans What is the highest level of education you plan Before Now Change1 to achieve? n % n % n % High School 156 17% 53 6% -103 -11.21% 2-year college 141 15% 118 13% -23 -2.30% 4-year college 243 27% 186 21% -57 -5.90% Graduate School 185 20% 253 28% +68 +7.97% Professional School 185 20% 284 32% +99 +11.44% Total 910 100% 894 100% Average2 2.91 3.35 0.000** (significant)3 1 Change from Before to Now. Increases are highlighted in green; decreases are highlighted in red. 2To compute averages, the following codes were applied: High School (1), 2-year college (2), 4-year college (3), Graduate School (4), Professional School (4). 3Paired samples t-test, p-value: **p<.01, *p<.05, p<.10 Prepared by The Findings Group, LLC Page 9 Table 10. Demographics Gender Female Male Total Ethnicity Asian Black Hispanic Native American White Multiracial Other Total n 445 467 912 n % 33 4% 327 36% 95 10% 4 0% 358 39% 75 8% 21 2% 913 100% Grade 6th 7th 8th 9th 10th 11th 12th Other Total % 49% 51% 100% n 170 218 235 46 50 78 112 5 914 % 19% 24% 26% 5% 5% 9% 12% 1% 100% Table 11. Participation How long have you participated in this program? n 0 semesters 4 1 semester 200 2 semesters 405 Dosage 3 semesters 4 or more semesters 40 143 Summer Only 1 Don't Know 120 Total 913 Did you participate in this program during the summer? n No 806 Summer Yes 50 Participation Don't Know 56 Total 912 % 0% 22% 44% 4% 16% 0% 13% 100% % 88% 5% 6% 100% Table 12. Program Rating Program Rating: How would you n rate this program? 911 Mean1 Assessment Very Poor (1) Poor (2) 4.20 Good +N+N+N+N+N+ 3% 1% 1 Reference lines are set at 3.5 and 4. Assessment: Good=Above 4.0; Attention=Below 4.0; Action=Below 3.5. Average (3) 15% Good (4) 34% Excellent (5) 47% Prepared by The Findings Group, LLC Page 10 Appendix A. Construct Reliabilities Construct Reliabilities Constructs n Intrinsic Motivation (9-items) Before 801 Now 782 Before 807 Self-Management/Self-Regulation (7-items) Now 784 Intent to Persist (5-items) Before 828 Now 817 Problem Solving (10-items) Now 816 Implementation Activities (5-items) Now 821 Note. Construct reliabilities were computed based on December 2012 data. Cronbach's alpha .853 .863 .729 .708 .861 .877 .848 .756 Reliability Interpretation Very good Very good Good Good Very good Very good Very good Good Cronbach's Alpha Reliability Key: Cronbach's alpha is a measure of the internal consistency of items in a construct. This statistic ranges from 0 to 1.00; the higher the value the better. An alpha of .80 or higher is considered to have achieved very good measurement reliability; an alpha of .65 is considered acceptable (Field, 2009). The table above suggests that all constructs achieved good to very good measurement reliability. Reliability Interpretation .90 and above Excellent reliability; at the level of the best measures .80 - .90 Very good .70 - .80 Good; in the range of most. There are probably a few items which could be improved. Somewhat low. This measure needs to be supplemented by other measures (e.g., .60 - .70 more surveys) to determine outcomes. There are probably some items which could be improved. .50 - .60 Suggests need for revision of measure, unless it is quite short (ten or fewer items). The test definitely needs to be supplemented by other measures (e.g., more tests). .50 or Questionable reliability. This measure should not contribute heavily to the outcomes below and needs revision. From: J. C. Nunnally, Psychometric Theory. New York: McGraw-Hill, 1967, pp. 172-235. Reference: Field, A. (2009). Discovering Statistics Using SPSS, 3rd Edition. Thousand Oaks, CA: Sage Publications. Prepared by The Findings Group, LLC Page 11