APPLIED LEARNING STUDENT QUESTIONNAIRE: ANALYSIS Overall Results December 2012 Executive Summary Participants and Methods In December 2012, 860 students across 6 Race to the Top programs completed the Applied Learning Student Questionnaire (ALSQ). The response rate displayed in Table 1 suggests that 95% of the total number of participating students were successfully surveyed. Table 1. Survey Response Rates Program STEM for Life Carroll County Drew Charter School- Partners of Innovation Murray County STEM Academy 21st Century STEM Collaboration- Barrow County STEM Targeted Education Program (STEP) Academy- Moore MS Tift County Mechatronics Program Total # of Survey Respondents 41 281 44 396 Total # of Participating Students 42 301 50 412 Survey Response Rate 98% 93% 88% 96% 62 64 97% 36 40 90% 860 909 95% The ALSQ1 is designed to measure pre and post gains related to student problem solving and communication skills, self-management and engagement. The ALSQ is a self-report questionnaire that includes 36 items to assess students' attitudes on the following survey constructs: 1. Intrinsic Motivation: motivation stemming from goals of mastery, learning and challenge. Example, "It is important for me to learn what is being taught in this program." 2. Self-management/Self-Regulation: effortful and persistent behaviors that are used to guide, monitor, and direct the success of one's learning and performance. Example, "I turn all my assignments in on time." 3. Intent to Persist: aspirations, plans, and goals to pursue additional education and a career in STEM. Example, "I intend to get a college degree in STEM (Science, Technology, Engineering, and Math)." 4. Problem Solving: inquiry-based learning environment that provides higher-order cognitive tasks and real-world applications. Example, "I work out explanations on my own." 5. Implementation Activities: hands-on activities designed to increase exposure to STEM topics and real-world applications. Example, "We learn what scientists/technicians/engineers/ mathematicians or other STEM professionals do." 1 See Appendix A for information related to the construct reliabilities of the ALSQ. Prepared by The Findings Group, LLC Page 1 Executive Summary, continued Results & Discussion ALSQ Survey Constructs Table 2 summarizes students' responses to the ALSQ survey constructs across all programs. It is clear that the programs were effective at producing statistically significantly increases in students' intrinsic motivation, self-management/self-regulation skills and intent to persist. The largest student gains observed were in the intrinsic motivation construct. Before the program, less than 55% of students indicated that they derive value and see the importance in learning about STEM; now, more than 76% say that they are intrinsically motivated to tackle STEM-related tasks and projects. Despite these statistically significant gains, it is important to note that the "now" scores across the following 3 constructs did not reach or exceed the optimal average of 4.0 on a 5-point likert scale (1, strongly disagree to 5, strongly agree): Intent to Persist, Problem Solving, and Implementation Activities. See Figure 1. In order to maximize effectiveness, we would expect students' average scores to exceed 4.0. Figure 1 suggests that additional work may be needed in the above mentioned areas. Table 2. Summary of Results by Constructs Constructs n Before 845 Intrinsic Motivation Now 834 Overall- Constructs Mean1 Paired Samples t-test 1 (Strongly Disagree) 2 (Disagree) 3 (Neutral) 4 (Agree) 5 (Strongly Agree) 3.57 0.000** 4.10 +N+N+N+N+N+ +N+N+N+N+N+ 5% 2% 11% 29% 31% 24% 4% 17% 34% 42% Self- Management/Self- Regulation Intent to Persist Before 840 Now 825 Before 841 Now 828 3.89 0.000** 4.13 +N+N+N+N+N+ +N+N+N+N+N+ 21% 21% 12% 8% 25% 25% 17% 19% 27% 26% 3.37 0.000** 3.64 +N+N+N+N+N+ +N+N+N+N+N+ 13% 10% 17% 13% 23% 15% 32% 20% 17% 40% Problem Solving Now 842 3.97 n/a +N+N+N+N+N+ 2% 5% 22% 36% 35% Implementation activities Now 838 3.77 n/a +N+N+N+N+N+ 3% 7% 26% 35% 28% 1 Reference lines are set at 3.5 and 4. Negatively worded statements were reverse coded for mean computations. **p<.01, *p<.05, p<.10 Figure 1. Constructs Before Now Optimal Average 5.00 4.50 ** 4.00 3.50 3.00 2.50 2.00 1, Strongly Disagree to 5, Strongly Agree Intrinsic Motivation Self- Management/Self- Regulation Intent to Persist Problem Solving Implementation Activitites **p<.01, *p<.05, p<.10; Scale is truncated for visual clarity. Prepared by The Findings Group, LLC Constructs Page 2 Executive Summary, continued ALSQ Survey Constructs by Program Examining the ALSQ results by individual program, it is evident that all programs show statistically significant increases in intrinsic motivation, self-management/self-regulation and intent to persist. Students in the STEM for Life program at Carroll County show the largest increases from before to now on all three of the above mentioned constructs; Murray STEM Academy students show the smallest average increases across all 6 programs. See Table 3. Table 3. Summary of Results by Constructs per Program Overall- Constructs per Program Constructs STEM for Life Carroll County (n=41) Drew Charter (n=281) Murray STEM Academy (n=44) 21st Century Barrow County (n=396) STEP Academy Moore MS (n=62) TIFT County Mechatronics (n=36) Mean t-test Mean t-test Mean t-test Mean t-test Mean t-test Mean t-test Intrinsic Motivation Before 3.27 Now 4.16 0.000** 3.69 4.12 0.000** 2.95 3.27 0.018* 3.59 4.15 0.000** 3.41 4.06 0.000** 3.83 4.31 0.000** Self- Before 3.42 3.97 Management/ Self-Regulation Now 4.19 0.000** 4.14 3.50 0.000** 3.61 0.010* 3.93 3.65 4.19 0.000** 4.00 4.09 0.000** 4.30 0.000** Intent to Persist Before 2.87 Now 3.49 0.000** 3.54 3.74 0.000** 2.98 3.06 0.499 3.32 3.58 0.000** 3.35 3.69 0.000** 3.74 4.26 0.000** Problem Solving Now 3.74 4.04 3.35 4.04 3.64 4.16 Implementation activities Now 3.70 n/a 3.76 n/a 2.96 3.94 n/a 3.43 n/a 4.23 n/a Scale= 1, Strongly Disagree to 5, Strongly Agree. Negatively worded statements were reverse coded for mean computations. **p<.01, *p<.05, p<.10 In order for programs to maximize their effectiveness, we would expect "now" scores to reach or exceed the optimal average of 4.0. Figures 2 6 display "now" scores for each program and construct. For example, Figure 2 indicates that all programs met or exceeded the optimal average for intrinsic motivation, with the exception of Murray STEM Academy. In general, programs not reaching or exceeding the red horizontal line may need additional attention. Figure 2. Intrinsic Motivation ("Now" Scores) 5 5 Figure 3. Self-Management/ Self-Regulation ("Now" Scores) 4.5 4.5 4 4 3.5 3.5 3 3 2.5 2.5 Scale= 1, Strongly Disagree to 5, Strongly Agree. Scale was truncated for visual clarity STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore MS TIFT County Mechatronics STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore MS TIFT County Mechatronics Prepared by The Findings Group, LLC Page 3 STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore MS TIFT County Mechatronics STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore MS TIFT County Mechatronics STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore MS TIFT County Mechatronics STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore MS TIFT County Mechatronics Executive Summary, continued 5 Figure 4. Intent to Persist ("Now" Scores) 4.5 4 3.5 3 2.5 5 Figure 6. Implementation Activities ("Now" Scores) 4.5 5 Figure 5. Problem Solving ("Now" Scores) 4.5 4 3.5 3 2.5 5 Figure 7. Program Ratings 4.5 4 4 3.5 3.5 3 3 2.5 2.5 Scale= 1, Strongly Disagree to 5, Strongly Agree. Scale was truncated for visual clarity; Program Rating Scale= 1, Very Poor to 5, Excellent. Program Rating Collapsing across all programs, students' ratings exceeded the optimal average of 4.0. On a 5-point likert scale where 1 signifies very poor and 5 signifies excellent, the average score was a 4.12. Looking at Figure 7, above, we see that 4 out of 6 programs were rated highly. However, Murray County STEM Academy and the STEP Academy at Moore Middle School may need additional assistance in improving student enjoyment. Areas for Further Improvement Across all programs, further enhancing problem solving skills may be warranted. Specifically, students' ratings suggest that the inquiry-based learning environment may be improved by allowing students more opportunity to choose their own topics or projects, work out explanations on their own, and plan and conduct their own projects. Likewise, encouraging programs to provide activities that foster interaction with STEM professionals may increase student exposure to real-world applications and careers. Such implementation activities may strengthen students' intentions and motivations to pursue additional education in STEM fields. Prepared by The Findings Group, LLC Page 4 Table 4. Intrinsic Motivation Intrinsic Motivation n 1. I prefer class work that is challenging so I can learn new things. Before 845 Now 834 2. It is important to me to learn what is being taught in this program. Before 844 Now 832 3. I like what I am learning in this program. Before 834 Now 828 4. I think I will be able to use what I learn in this program in other classes. Before 831 Now 828 5. Even when I do poorly on a test, Before 841 I try to learn from my mistakes. Now 831 6. I think that what I am learning in Before 833 this program is useful for me to know. Now 826 7. I think that what we are learning Before 831 in this program is interesting. Now 825 8. Understanding STEM (Science, Technology, Engineering, and Math) is important to me. Before 840 Now 831 9. I enjoy STEM (Science, Before 834 Technology, Engineering, and Math) in general. Now 828 1 Reference lines are set at 3.5 and 4. **p<.01, *p<.05, p<.10 Mean1 Paired Samples t-test 1 (Strongly Disagree) 2 3 (Disagree) (Neutral) 4 (Agree) 5 (Strongly Agree) 3.07 +N+N+N+N+N+ 10% 3.68 0.000** +N+N+N+N+N+ 4% 17% 38% 24% 10% 8% 26% 39% 23% 3.85 +N+N+N+N+N+ 2% 4.31 0.000** +N+N+N+N+N+ 1% 5% 27% 38% 28% 1% 12% 37% 49% 3.55 +N+N+N+N+N+ 5% 4.12 0.000** +N+N+N+N+N+ 3% 3.57 +N+N+N+N+N+ 5% 4.10 0.000** +N+N+N+N+N+ 2% 11% 32% 30% 23% 4% 17% 33% 44% 11% 29% 33% 22% 5% 17% 33% 43% 4.00 +N+N+N+N+N+ 2% 4.44 0.000** +N+N+N+N+N+ 1% 3.70 +N+N+N+N+N+ 3% 4.21 0.000** +N+N+N+N+N+ 1% 7% 18% 35% 38% 1% 9% 32% 58% 9% 25% 37% 24% 3% 15% 35% 46% 3.47 +N+N+N+N+N+ 7% 4.01 0.000** +N+N+N+N+N+ 3% 3.60 +N+N+N+N+N+ 5% 4.13 0.000** +N+N+N+N+N+ 2% 13% 31% 26% 24% 5% 19% 31% 41% 9% 31% 30% 25% 4% 18% 32% 44% 3.32 +N+N+N+N+N+ 9% 3.85 0.000** +N+N+N+N+N+ 5% 13% 31% 28% 18% 6% 22% 34% 33% Prepared by The Findings Group, LLC Page 5 Table 5. Self-Regulation/Self-Motivation Self-Regulation/Self-Motivation n Mean1 Paired Samples t-test 1 (Strongly Disagree) 2 3 (Disagree) (Neutral) 4 (Agree) 5 (Strongly Agree) 10. I turn all my assignments in on Before 840 time. Now 818 3.51 +N+N+N+N+N+ 4% 3.91 0.000** +N+N+N+N+N+ 1% 13% 33% 31% 20% 6% 25% 38% 30% 11. I miss class often. (n) Before 835 Now 817 1.56 +N+N+N+N+N+ 66% 20% 9% 4% 2% 1.54 0.640 +N+N+N+N+N+ 71% 15% 7% 4% 3% 12. I am often late for class. (n) Before 828 Now 814 1.57 +N+N+N+N+N+ 65% 21% 9% 4% 1% 1.52 0.045* +N+N+N+N+N+ 69% 18% 8% 3% 2% 13. I set aside time to do my homework and study. Before 838 Now 825 3.34 +N+N+N+N+N+ 6% 3.72 0.000** +N+N+N+N+N+ 4% 15% 34% 30% 15% 7% 29% 32% 27% 14. When I say I'm going to do something, I do it. Before 840 Now 823 3.79 +N+N+N+N+N+ 2% 4.05 0.000** +N+N+N+N+N+ 1% 5% 31% 37% 25% 3% 21% 39% 36% 15. I am a hard worker. Before 836 Now 823 4.00 +N+N+N+N+N+ 1% 4.30 0.000** +N+N+N+N+N+ 0% 4% 24% 36% 35% 1% 15% 34% 49% 16. I finish whatever I begin. Before 837 Now 822 3.69 +N+N+N+N+N+ 2% 4.01 0.000** +N+N+N+N+N+ 0% 1 Reference lines are set at 3.5 and 4. **p<.01, *p<.05, p<.10; (n) nega vely worded statement 8% 35% 32% 24% 3% 25% 37% 34% Prepared by The Findings Group, LLC Page 6 Table 6. Intent to Persist Intent to Persist n 17. I am considering a career in STEM (Science, Technology, Engineering, and Math). Before 841 Now 827 18. I intend to get a college degree Before 839 in STEM (Science, Technology, Engineering, and Math). Now 826 19. I can see myself working in STEM (Science, Technology, Engineering, and Math). Before 836 Now 825 20. Someday, I would like to have a Before 833 career in STEM (Science, Technology, Engineering, and Math). Now 819 21. I intend to graduate from high Before 835 school Now 828 1 Reference lines are set at 3.5 and 4. **p<.01, *p<.05, p<.10 Mean1 Paired Samples t-test 1 (Strongly Disagree) 2 3 (Disagree) (Neutral) 4 (Agree) 5 (Strongly Agree) 3.00 +N+N+N+N+N+ 16% 3.32 0.000** +N+N+N+N+N+ 13% 21% 29% 17% 18% 17% 23% 21% 27% 3.17 +N+N+N+N+N+ 11% 3.47 0.000** +N+N+N+N+N+ 10% 20% 29% 18% 21% 14% 27% 18% 31% 2.99 +N+N+N+N+N+ 17% 3.31 0.000** +N+N+N+N+N+ 13% 19% 28% 19% 17% 16% 24% 22% 25% 2.92 +N+N+N+N+N+ 19% 21% 27% 17% 17% 3.23 0.000** +N+N+N+N+N+ 15% 16% 25% 18% 26% 4.79 +N+N+N+N+N+ 1% 4.85 0.002** +N+N+N+N+N+ 1% 1% 4% 6% 88% 1% 2% 4% 92% Prepared by The Findings Group, LLC Page 7 Table 7. Problem Solving, Now Only 1 Problem Solving n Mean Assessment (Strongly Disagree) 22. In this program, my teacher(s) tells me how to improve my 838 4.18 Good +N+N+N+N+N+ 2% work. 23. In this program, my teacher(s) lets us choose our own topics 838 3.44 Action ! +N+N+N+N+N+ 7% or projects to investigate. 24. In this program, I work out explanations on my own. 841 3.65 Attention +N+N+N+N+N+ 1% 25. In this program, I have opportunities to explain my 842 3.87 Attention +N+N+N+N+N+ 2% ideas. 26. In this program, we plan and do our own projects and/or 842 3.76 Attention +N+N+N+N+N+ 3% experiments. 27. In this program, we work on real-world problems. 840 3.90 Attention +N+N+N+N+N+ 3% 28. In this program, we have class discussions. 836 4.27 Good +N+N+N+N+N+ 2% 29. In this program, we investigate to see if our ideas are right. 838 4.05 Good +N+N+N+N+N+ 1% 30. In this program, we need to be able to think and ask questions. 834 4.35 Good +N+N+N+N+N+ 1% 31. In this program, we are expected to understand and 837 4.23 Good +N+N+N+N+N+ 1% explain ideas. 1 Reference lines are set at 3.5 and 4. Assessment: Good=Above 4.0; Attention=Below 4.0; Action=Below 3.5. 2 (Disagree) 3% 11% 5% 6% 8% 7% 3% 3% 2% 2% 3 (Neutral) 14% 32% 36% 25% 27% 22% 13% 22% 13% 16% 4 (Agree) 40% 32% 41% 38% 33% 34% 32% 36% 32% 37% 5 (Strongly Agree) 42% 18% 16% 29% 29% 34% 51% 37% 53% 45% Prepared by The Findings Group, LLC Page 8 Table 8. Implementation Activities, Now Only 1 Implementation Activities n Mean Assessment (Strongly Disagree) 32. In this program, my teacher(s) takes notice of 834 3.93 Attention +N+N+N+N+N+ 3% students' ideas. 33. In this program, my teacher(s) shows us how new information relates to what 828 4.18 Good +N+N+N+N+N+ 1% we have already learned. 34. In this program, we learn what scientists/ technicians/ engineers/ mathematicians 838 3.58 Attention +N+N+N+N+N+ 4% or other STEM professionals do. 35. In this program, we do our work in groups. 837 3.71 Attention +N+N+N+N+N+ 2% 36. In this program, we interact with scientists/ technicians/ engineers/ mathematicians 835 3.45 Action ! +N+N+N+N+N+ 8% or other STEM professionals. 1 Reference lines are set at 3.5 and 4. Assessment: Good=Above 4.0; Attention=Below 4.0; Action=Below 3.5. 2 (Disagree) 5% 3% 10% 7% 12% 3 (Neutral) 22% 15% 30% 33% 30% 4 (Agree) 39% 37% 35% 35% 28% 5 (Strongly Agree) 32% 43% 21% 23% 22% Table 9. Educational Plans What is the highest level of education you plan Before Now Change1 to achieve? n % n % n % High School 104 13% 69 9% -35 -4.16% 2-year college 103 13% 61 8% -42 -5.04% 4-year college 250 31% 168 21% -82 -9.72% Graduate School 180 22% 226 28% +46 +6.20% Professional School 174 21% 272 34% +98 +12.72% Total 811 100% 796 100% Average2 3.05 3.38 0.000** (significant)3 1 Change from Before to Now. Increases are highlighted in green; decreases are highlighted in red. 2To compute averages, the following codes were applied: High School (1), 2-year college (2), 4-year college (3), Graduate School (4), Professional School (4). 3Paired samples t-test, p-value: **p<.01, *p<.05, p<.10 Prepared by The Findings Group, LLC Page 9 Table 10. Demographics Gender Female Male Total Ethnicity Asian Black Hispanic Native American White Multiracial Other Total n 436 406 842 n % 31 4% 316 38% 69 8% 4 0% 337 40% 64 8% 19 2% 840 100% Grade 6th 7th 8th 9th 10th 11th 12th Other Total % 52% 48% 100% n 178 240 220 47 47 27 86 0 845 % 21% 28% 26% 6% 6% 3% 10% 0% 100% Table 11. Participation How long have you participated in this program? n 0 semesters 34 1 semester 515 2 semesters 67 Dosage 3 semesters 4 or more semesters 16 84 Summer Only 2 Don't Know 114 Total 832 Did you participate in this program during the summer? n No 720 Summer Yes 41 Participation Don't Know 73 Total 834 % 4% 62% 8% 2% 10% 0% 14% 100% % 86% 5% 9% 100% Table 12. Program Rating Program Rating: How would you n rate this program? 836 Mean1 Assessment Very Poor (1) Poor (2) 4.12 Good +N+N+N+N+N+ 3% 3% 1 Reference lines are set at 3.5 and 4. Assessment: Good=Above 4.0; Attention=Below 4.0; Action=Below 3.5. Average (3) 17% Good (4) 35% Excellent (5) 42% Prepared by The Findings Group, LLC Page 10 Appendix A. Construct Reliabilities Construct Reliabilities Constructs n Intrinsic Motivation (9-items) Before 801 Now 782 Before 807 Self-Management/Self-Regulation (7-items) Now 784 Intent to Persist (5-items) Before 828 Now 817 Problem Solving (10-items) Now 816 Implementation Activities (5-items) Now 821 Cronbach's alpha .853 .863 .729 .708 .861 .877 .848 .756 Reliability Interpretation Very good Very good Good Good Very good Very good Very good Good Cronbach's Alpha Reliability Key: Cronbach's alpha is a measure of the internal consistency of items in a construct. This statistic ranges from 0 to 1.00; the higher the value the better. An alpha of .80 or higher is considered to have achieved very good measurement reliability; an alpha of .65 is considered acceptable (Field, 2009). The table above suggests that all constructs achieved good to very good measurement reliability. Reliability Interpretation .90 and above Excellent reliability; at the level of the best measures .80 - .90 Very good .70 - .80 Good; in the range of most. There are probably a few items which could be improved. Somewhat low. This measure needs to be supplemented by other measures (e.g., .60 - .70 more surveys) to determine outcomes. There are probably some items which could be improved. .50 - .60 Suggests need for revision of measure, unless it is quite short (ten or fewer items). The test definitely needs to be supplemented by other measures (e.g., more tests). .50 or Questionable reliability. This measure should not contribute heavily to the outcomes below and needs revision. From: J. C. Nunnally, Psychometric Theory. New York: McGraw-Hill, 1967, pp. 172-235. Reference: Field, A. (2009). Discovering Statistics Using SPSS, 3rd Edition. Thousand Oaks, CA: Sage Publications. Prepared by The Findings Group, LLC Page 11