Applied learning student questionnaire analysis overall results

APPLIED LEARNING STUDENT QUESTIONNAIRE: ANALYSIS

Overall Results
December 2012
Executive Summary
Participants and Methods In December 2012, 860 students across 6 Race to the Top programs completed the Applied Learning Student Questionnaire (ALSQ). The response rate displayed in Table 1 suggests that 95% of the total number of participating students were successfully surveyed.

Table 1. Survey Response Rates
Program
STEM for Life Carroll County Drew Charter School- Partners of Innovation Murray County STEM Academy 21st Century STEM Collaboration- Barrow County STEM Targeted Education Program (STEP) Academy- Moore MS Tift County Mechatronics Program Total

# of Survey Respondents
41 281 44
396

Total # of Participating Students 42 301 50
412

Survey Response Rate
98% 93% 88%
96%

62

64

97%

36

40

90%

860

909

95%

The ALSQ1 is designed to measure pre and post gains related to student problem solving and communication skills, self-management and engagement.
The ALSQ is a self-report questionnaire that includes 36 items to assess students' attitudes on the following survey constructs:
1. Intrinsic Motivation: motivation stemming from goals of mastery, learning and challenge. Example, "It is important for me to learn what is being taught in this program."
2. Self-management/Self-Regulation: effortful and persistent behaviors that are used to guide, monitor, and direct the success of one's learning and performance. Example, "I turn all my assignments in on time."
3. Intent to Persist: aspirations, plans, and goals to pursue additional education and a career in STEM. Example, "I intend to get a college degree in STEM (Science, Technology, Engineering, and Math)."
4. Problem Solving: inquiry-based learning environment that provides higher-order cognitive tasks and real-world applications. Example, "I work out explanations on my own."
5. Implementation Activities: hands-on activities designed to increase exposure to STEM topics and real-world applications. Example, "We learn what scientists/technicians/engineers/ mathematicians or other STEM professionals do."

1 See Appendix A for information related to the construct reliabilities of the ALSQ.
Prepared by The Findings Group, LLC

Page 1

Executive Summary, continued

Results & Discussion

ALSQ Survey Constructs Table 2 summarizes students' responses to the ALSQ survey constructs across all programs. It is clear that the programs were effective at producing statistically significantly increases in students' intrinsic motivation, self-management/self-regulation skills and intent to persist. The largest student gains observed were in the intrinsic motivation construct. Before the program, less than 55% of students indicated that they derive value and see the importance in learning about STEM; now, more than 76% say that they are intrinsically motivated to tackle STEM-related tasks and projects. Despite these statistically significant gains, it is important to note that the "now" scores across the following 3 constructs did not reach or exceed the optimal average of 4.0 on a 5-point likert scale (1, strongly disagree to 5, strongly agree): Intent to Persist, Problem Solving, and Implementation Activities. See Figure 1. In order to maximize effectiveness, we would expect students' average scores to exceed 4.0. Figure 1 suggests that additional work may be needed in the above mentioned areas.

Table 2. Summary of Results by Constructs

Constructs

n

Before 845 Intrinsic Motivation
Now 834

Overall- Constructs

Mean1

Paired Samples
t-test

1 (Strongly Disagree)

2 (Disagree)

3 (Neutral)

4 (Agree)

5 (Strongly
Agree)

3.57 0.000** 4.10

+N+N+N+N+N+ +N+N+N+N+N+

5% 2%

11% 29% 31% 24% 4% 17% 34% 42%

Self- Management/Self-
Regulation
Intent to Persist

Before 840 Now 825 Before 841 Now 828

3.89 0.000** 4.13

+N+N+N+N+N+ +N+N+N+N+N+

21% 21%

12% 8%

25% 25% 17% 19% 27% 26%

3.37 0.000** 3.64

+N+N+N+N+N+ +N+N+N+N+N+

13% 10%

17% 13%

23% 15% 32% 20% 17% 40%

Problem Solving Now 842

3.97 n/a +N+N+N+N+N+ 2% 5% 22% 36% 35%

Implementation activities

Now 838

3.77 n/a +N+N+N+N+N+ 3% 7% 26% 35% 28%

1 Reference lines are set at 3.5 and 4. Negatively worded statements were reverse coded for mean computations. **p<.01, *p<.05, p<.10

Figure 1. Constructs

Before

Now

Optimal Average

5.00

4.50 **
4.00

3.50

3.00

2.50

2.00

1, Strongly Disagree to 5, Strongly Agree Intrinsic Motivation
Self- Management/Self-
Regulation Intent to Persist Problem Solving Implementation
Activitites

**p<.01, *p<.05, p<.10; Scale is truncated for visual clarity. Prepared by The Findings Group, LLC

Constructs

Page 2

Executive Summary, continued
ALSQ Survey Constructs by Program Examining the ALSQ results by individual program, it is evident that all programs show statistically significant increases in intrinsic motivation, self-management/self-regulation and intent to persist. Students in the STEM for Life program at Carroll County show the largest increases from before to now on all three of the above mentioned constructs; Murray STEM Academy students show the smallest average increases across all 6 programs. See Table 3.

Table 3. Summary of Results by Constructs per Program

Overall- Constructs per Program

Constructs

STEM for Life Carroll
County (n=41)

Drew Charter
(n=281)

Murray STEM Academy
(n=44)

21st Century Barrow County
(n=396)

STEP Academy Moore MS
(n=62)

TIFT County Mechatronics
(n=36)

Mean t-test Mean t-test Mean t-test Mean t-test Mean t-test Mean t-test

Intrinsic Motivation

Before 3.27 Now 4.16

0.000**

3.69 4.12

0.000**

2.95 3.27

0.018*

3.59 4.15

0.000**

3.41 4.06

0.000**

3.83 4.31

0.000**

Self-

Before 3.42

3.97

Management/ Self-Regulation

Now 4.19 0.000** 4.14

3.50 0.000** 3.61

0.010*

3.93

3.65

4.19 0.000** 4.00

4.09 0.000** 4.30

0.000**

Intent to Persist

Before 2.87 Now 3.49

0.000**

3.54 3.74

0.000**

2.98 3.06

0.499

3.32 3.58

0.000**

3.35 3.69

0.000**

3.74 4.26

0.000**

Problem Solving Now 3.74

4.04

3.35

4.04

3.64

4.16

Implementation activities

Now 3.70

n/a

3.76

n/a 2.96

3.94 n/a 3.43 n/a 4.23

n/a

Scale= 1, Strongly Disagree to 5, Strongly Agree. Negatively worded statements were reverse coded for mean computations. **p<.01,

*p<.05, p<.10

In order for programs to maximize their effectiveness, we would expect "now" scores to reach or

exceed the optimal average of 4.0. Figures 2 6 display "now" scores for each program and

construct. For example, Figure 2 indicates that all programs met or exceeded the optimal average

for intrinsic motivation, with the exception of Murray STEM Academy. In general, programs not

reaching or exceeding the red horizontal line may need additional attention.

Figure 2. Intrinsic Motivation ("Now" Scores)
5

5

Figure 3. Self-Management/ Self-Regulation ("Now" Scores)

4.5

4.5

4

4

3.5

3.5

3
3

2.5
2.5
Scale= 1, Strongly Disagree to 5, Strongly Agree. Scale was truncated for visual clarity

STEM for Life Carroll County Drew Charter
Murray STEM Academy 21st Century Barrow County STEP Academy Moore MS TIFT County Mechatronics
STEM for Life Carroll County Drew Charter
Murray STEM Academy 21st Century Barrow County STEP Academy Moore MS TIFT County Mechatronics

Prepared by The Findings Group, LLC

Page 3

STEM for Life Carroll County
Drew Charter Murray STEM
Academy 21st Century Barrow
County STEP Academy
Moore MS TIFT County Mechatronics
STEM for Life Carroll County Drew Charter
Murray STEM Academy 21st Century Barrow County STEP Academy Moore MS TIFT County Mechatronics

STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow
County STEP Academy
Moore MS TIFT County Mechatronics
STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore MS
TIFT County Mechatronics

Executive Summary, continued
5
Figure 4. Intent to Persist ("Now" Scores)
4.5
4
3.5
3
2.5 5
Figure 6. Implementation Activities ("Now" Scores)
4.5

5
Figure 5. Problem Solving ("Now" Scores)
4.5

4

3.5

3

2.5

5

Figure 7. Program Ratings

4.5

4

4

3.5

3.5

3

3

2.5

2.5

Scale= 1, Strongly Disagree to 5, Strongly Agree. Scale was truncated for visual clarity; Program Rating Scale= 1, Very Poor to 5, Excellent.

Program Rating Collapsing across all programs, students' ratings exceeded the optimal average of 4.0. On a 5-point likert scale where 1 signifies very poor and 5 signifies excellent, the average score was a 4.12. Looking at Figure 7, above, we see that 4 out of 6 programs were rated highly. However, Murray County STEM Academy and the STEP Academy at Moore Middle School may need additional assistance in improving student enjoyment.
Areas for Further Improvement Across all programs, further enhancing problem solving skills may be warranted. Specifically, students' ratings suggest that the inquiry-based learning environment may be improved by allowing students more opportunity to choose their own topics or projects, work out explanations on their own, and plan and conduct their own projects. Likewise, encouraging programs to provide activities that foster interaction with STEM professionals may increase student exposure to real-world applications and careers. Such implementation activities may strengthen students' intentions and motivations to pursue additional education in STEM fields.

Prepared by The Findings Group, LLC

Page 4

Table 4. Intrinsic Motivation

Intrinsic Motivation

n

1. I prefer class work that is challenging so I can learn new things.

Before 845 Now 834

2. It is important to me to learn what is being taught in this program.

Before 844 Now 832

3. I like what I am learning in this program.

Before 834 Now 828

4. I think I will be able to use what I learn in this program in other classes.

Before 831 Now 828

5. Even when I do poorly on a test, Before 841

I try to learn from my mistakes. Now 831

6. I think that what I am learning in Before 833

this program is useful for me to

know.

Now 826

7. I think that what we are learning Before 831

in this program is interesting.

Now 825

8. Understanding STEM (Science, Technology, Engineering, and Math) is important to me.

Before 840 Now 831

9. I enjoy STEM (Science,

Before 834

Technology, Engineering, and

Math) in general.

Now 828

1 Reference lines are set at 3.5 and 4. **p<.01, *p<.05, p<.10

Mean1

Paired Samples
t-test

1 (Strongly Disagree)

2

3

(Disagree) (Neutral)

4 (Agree)

5 (Strongly Agree)

3.07

+N+N+N+N+N+ 10%

3.68 0.000** +N+N+N+N+N+ 4%

17%

38% 24% 10%

8%

26% 39% 23%

3.85

+N+N+N+N+N+ 2%

4.31 0.000** +N+N+N+N+N+ 1%

5%

27% 38% 28%

1%

12% 37% 49%

3.55

+N+N+N+N+N+ 5%

4.12 0.000** +N+N+N+N+N+ 3%

3.57

+N+N+N+N+N+ 5%

4.10 0.000** +N+N+N+N+N+ 2%

11%

32% 30% 23%

4%

17% 33% 44%

11%

29% 33% 22%

5%

17% 33% 43%

4.00

+N+N+N+N+N+ 2%

4.44 0.000** +N+N+N+N+N+ 1%

3.70

+N+N+N+N+N+ 3%

4.21 0.000** +N+N+N+N+N+ 1%

7%

18% 35% 38%

1%

9%

32% 58%

9%

25% 37% 24%

3%

15% 35% 46%

3.47

+N+N+N+N+N+ 7%

4.01 0.000** +N+N+N+N+N+ 3%

3.60

+N+N+N+N+N+ 5%

4.13 0.000** +N+N+N+N+N+ 2%

13%

31% 26% 24%

5%

19% 31% 41%

9%

31% 30% 25%

4%

18% 32% 44%

3.32

+N+N+N+N+N+ 9%

3.85 0.000** +N+N+N+N+N+ 5%

13%

31% 28% 18%

6%

22% 34% 33%

Prepared by The Findings Group, LLC

Page 5

Table 5. Self-Regulation/Self-Motivation

Self-Regulation/Self-Motivation

n

Mean1

Paired Samples
t-test

1 (Strongly Disagree)

2

3

(Disagree) (Neutral)

4 (Agree)

5 (Strongly Agree)

10. I turn all my assignments in on Before 840

time.

Now 818

3.51

+N+N+N+N+N+ 4%

3.91 0.000** +N+N+N+N+N+ 1%

13%

33% 31% 20%

6%

25% 38% 30%

11. I miss class often. (n)

Before 835 Now 817

1.56

+N+N+N+N+N+ 66%

20%

9%

4%

2%

1.54 0.640 +N+N+N+N+N+ 71%

15%

7%

4%

3%

12. I am often late for class. (n)

Before 828 Now 814

1.57

+N+N+N+N+N+ 65%

21%

9%

4%

1%

1.52 0.045* +N+N+N+N+N+ 69%

18%

8%

3%

2%

13. I set aside time to do my homework and study.

Before 838 Now 825

3.34

+N+N+N+N+N+ 6%

3.72 0.000** +N+N+N+N+N+ 4%

15%

34% 30% 15%

7%

29% 32% 27%

14. When I say I'm going to do something, I do it.

Before 840 Now 823

3.79

+N+N+N+N+N+ 2%

4.05 0.000** +N+N+N+N+N+ 1%

5%

31% 37% 25%

3%

21% 39% 36%

15. I am a hard worker.

Before 836 Now 823

4.00

+N+N+N+N+N+ 1%

4.30 0.000** +N+N+N+N+N+ 0%

4%

24% 36% 35%

1%

15% 34% 49%

16. I finish whatever I begin.

Before 837 Now 822

3.69

+N+N+N+N+N+ 2%

4.01 0.000** +N+N+N+N+N+ 0%

1 Reference lines are set at 3.5 and 4. **p<.01, *p<.05, p<.10; (n) nega vely worded statement

8%

35% 32% 24%

3%

25% 37% 34%

Prepared by The Findings Group, LLC

Page 6

Table 6. Intent to Persist

Intent to Persist

n

17. I am considering a career in STEM (Science, Technology, Engineering, and Math).

Before 841 Now 827

18. I intend to get a college degree Before 839

in STEM (Science, Technology, Engineering, and Math).

Now 826

19. I can see myself working in STEM (Science, Technology, Engineering, and Math).

Before 836 Now 825

20. Someday, I would like to have a Before 833 career in STEM (Science,

Technology, Engineering, and Math).

Now 819

21. I intend to graduate from high Before 835

school

Now 828

1 Reference lines are set at 3.5 and 4. **p<.01, *p<.05, p<.10

Mean1

Paired Samples
t-test

1 (Strongly Disagree)

2

3

(Disagree) (Neutral)

4 (Agree)

5 (Strongly Agree)

3.00

+N+N+N+N+N+ 16%

3.32 0.000** +N+N+N+N+N+ 13%

21%

29% 17% 18%

17%

23% 21% 27%

3.17

+N+N+N+N+N+ 11%

3.47 0.000** +N+N+N+N+N+ 10%

20%

29% 18% 21%

14%

27% 18% 31%

2.99

+N+N+N+N+N+ 17%

3.31 0.000** +N+N+N+N+N+ 13%

19%

28% 19% 17%

16%

24% 22% 25%

2.92

+N+N+N+N+N+ 19%

21%

27% 17% 17%

3.23 0.000** +N+N+N+N+N+ 15%

16%

25% 18% 26%

4.79

+N+N+N+N+N+ 1%

4.85 0.002** +N+N+N+N+N+ 1%

1%

4%

6%

88%

1%

2%

4%

92%

Prepared by The Findings Group, LLC

Page 7

Table 7. Problem Solving, Now Only

1

Problem Solving

n

Mean

Assessment

(Strongly

Disagree)

22. In this program, my teacher(s)

tells me how to improve my

838

4.18 Good +N+N+N+N+N+ 2%

work.

23. In this program, my teacher(s) lets us choose our own topics 838

3.44 Action ! +N+N+N+N+N+ 7%

or projects to investigate.

24. In this program, I work out explanations on my own.

841

3.65 Attention +N+N+N+N+N+ 1%

25. In this program, I have

opportunities to explain my

842

3.87 Attention +N+N+N+N+N+ 2%

ideas.

26. In this program, we plan and do

our own projects and/or

842

3.76 Attention +N+N+N+N+N+ 3%

experiments.

27. In this program, we work on real-world problems.

840

3.90 Attention +N+N+N+N+N+ 3%

28. In this program, we have class discussions.

836

4.27 Good +N+N+N+N+N+ 2%

29. In this program, we investigate to see if our ideas are right.

838

4.05 Good +N+N+N+N+N+ 1%

30. In this program, we need to be able to think and ask questions.

834

4.35 Good +N+N+N+N+N+ 1%

31. In this program, we are expected to understand and 837

4.23 Good +N+N+N+N+N+ 1%

explain ideas. 1 Reference lines are set at 3.5 and 4. Assessment: Good=Above 4.0; Attention=Below 4.0; Action=Below 3.5.

2 (Disagree)
3%
11% 5% 6%
8% 7% 3% 3% 2% 2%

3 (Neutral)
14%
32% 36% 25%
27% 22% 13% 22% 13% 16%

4 (Agree)
40%
32% 41% 38%
33% 34% 32% 36% 32% 37%

5 (Strongly
Agree) 42%
18% 16% 29%
29% 34% 51% 37% 53% 45%

Prepared by The Findings Group, LLC

Page 8

Table 8. Implementation Activities, Now Only

1

Implementation Activities

n

Mean

Assessment

(Strongly

Disagree)

32. In this program, my teacher(s) takes notice of 834

3.93 Attention +N+N+N+N+N+ 3%

students' ideas.

33. In this program, my

teacher(s) shows us how new information relates to what

828

4.18 Good +N+N+N+N+N+ 1%

we have already learned.

34. In this program, we learn

what scientists/ technicians/ engineers/ mathematicians 838

3.58 Attention +N+N+N+N+N+ 4%

or other STEM professionals

do.

35. In this program, we do our work in groups.

837

3.71 Attention +N+N+N+N+N+ 2%

36. In this program, we interact

with scientists/ technicians/ engineers/ mathematicians

835

3.45 Action ! +N+N+N+N+N+ 8%

or other STEM professionals. 1 Reference lines are set at 3.5 and 4. Assessment: Good=Above 4.0; Attention=Below 4.0; Action=Below 3.5.

2 (Disagree)
5% 3%
10% 7% 12%

3 (Neutral)
22% 15%
30% 33% 30%

4 (Agree)
39% 37%
35% 35% 28%

5 (Strongly Agree)
32%
43%
21%
23% 22%

Table 9. Educational Plans

What is the highest level of education you plan

Before

Now

Change1

to achieve?

n

%

n

%

n

%

High School

104 13%

69

9%

-35

-4.16%

2-year college

103 13%

61

8%

-42

-5.04%

4-year college

250 31%

168

21%

-82

-9.72%

Graduate School

180 22%

226

28%

+46

+6.20%

Professional School

174 21%

272

34%

+98

+12.72%

Total

811 100% 796

100%

Average2

3.05

3.38

0.000** (significant)3

1 Change from Before to Now. Increases are highlighted in green; decreases are highlighted in red.

2To compute averages, the following codes were applied: High School (1), 2-year college (2), 4-year college (3), Graduate School (4), Professional School (4). 3Paired

samples t-test, p-value: **p<.01, *p<.05, p<.10

Prepared by The Findings Group, LLC

Page 9

Table 10. Demographics
Gender
Female Male Total
Ethnicity
Asian Black Hispanic Native American White Multiracial Other Total

n 436 406 842

n

%

31

4%

316

38%

69

8%

4

0%

337

40%

64

8%

19

2%

840 100%

Grade
6th 7th 8th 9th 10th 11th 12th
Other
Total

% 52% 48% 100%
n 178 240 220 47 47 27 86
0 845

% 21% 28% 26% 6% 6% 3% 10% 0% 100%

Table 11. Participation

How long have you participated in this program?

n

0 semesters

34

1 semester

515

2 semesters

67

Dosage

3 semesters 4 or more semesters

16 84

Summer Only

2

Don't Know

114

Total

832

Did you participate in this program during the summer? n

No

720

Summer

Yes

41

Participation Don't Know

73

Total

834

% 4% 62% 8% 2% 10% 0% 14% 100%
% 86% 5% 9% 100%

Table 12. Program Rating

Program Rating: How would you n
rate this program? 836

Mean1

Assessment

Very Poor (1)

Poor (2)

4.12 Good +N+N+N+N+N+ 3% 3%

1 Reference lines are set at 3.5 and 4. Assessment: Good=Above 4.0; Attention=Below 4.0; Action=Below 3.5.

Average (3)
17%

Good (4)
35%

Excellent (5)
42%

Prepared by The Findings Group, LLC

Page 10

Appendix A. Construct Reliabilities

Construct Reliabilities

Constructs

n

Intrinsic Motivation (9-items)

Before

801

Now

782

Before

807

Self-Management/Self-Regulation (7-items)

Now

784

Intent to Persist (5-items)

Before

828

Now

817

Problem Solving (10-items)

Now

816

Implementation Activities (5-items)

Now

821

Cronbach's alpha
.853 .863 .729 .708 .861 .877 .848 .756

Reliability Interpretation
Very good Very good
Good Good Very good Very good Very good Good

Cronbach's Alpha Reliability Key: Cronbach's alpha is a measure of the internal consistency of items in a construct. This statistic ranges from 0 to 1.00; the higher the value the better. An alpha of .80 or higher is considered to have achieved very good measurement reliability; an alpha of .65 is considered acceptable (Field, 2009). The table above suggests that all constructs achieved good to very good measurement reliability.

Reliability

Interpretation

.90 and above

Excellent reliability; at the level of the best measures

.80 - .90 Very good

.70 - .80 Good; in the range of most. There are probably a few items which could be improved.

Somewhat low. This measure needs to be supplemented by other measures (e.g., .60 - .70 more surveys) to determine outcomes. There are probably some items which could be
improved.

.50 - .60

Suggests need for revision of measure, unless it is quite short (ten or fewer items). The test definitely needs to be supplemented by other measures (e.g., more tests).

.50 or Questionable reliability. This measure should not contribute heavily to the outcomes below and needs revision.
From: J. C. Nunnally, Psychometric Theory. New York: McGraw-Hill, 1967, pp. 172-235.

Reference: Field, A. (2009). Discovering Statistics Using SPSS, 3rd Edition. Thousand Oaks, CA: Sage Publications.

Prepared by The Findings Group, LLC

Page 11