Applied learning student questionnaire analysis overall results

APPLIED LEARNING STUDENT
QUESTIONNAIRE: OVERALL
ANALYSIS
Overall Results May 2013
Executive Summary
Participants and Methods In May 2013, 928 students across 6 Race to the Top programs completed the Applied Learning Student Questionnaire (ALSQ). The response rates displayed in Table 1 suggest that 90% of the total number of participating students were successfully surveyed.

Table 1. Survey Response Rates
Program
STEM for Life Carroll County Drew Charter School- Partners of Innovation Murray County STEM Academy 21st Century STEM Collaboration- Barrow County STEM Targeted Education Program (STEP) Academy- Sweetwater MS and Moore MS Tift County Mechatronics Program Total

# of Survey Respondents
160 273 37
318

Total # of Participating Students 160 301 50
365

Survey Response Rate
100% 91% 74%
87%

109

121

90%

31

36

86%

928

1033

90%

The ALSQ1 is designed to measure pre and post gains related to student problem solving and communication skills, self-management and engagement.
The ALSQ is a self-report questionnaire that includes 36 items to assess students' attitudes on the following survey constructs:
1. Intrinsic Motivation: motivation stemming from goals of mastery, learning and challenge. Example, "It is important for me to learn what is being taught in this program."
2. Self-management/Self-Regulation: effortful and persistent behaviors that are used to guide, monitor, and direct the success of one's learning and performance. Example, "I turn all my assignments in on time."
3. Intent to Persist: aspirations, plans, and goals to pursue additional education and a career in STEM. Example, "I intend to get a college degree in STEM (Science, Technology, Engineering, and Math)."
4. Problem Solving: inquiry-based learning environment that provides higher-order cognitive tasks and real-world applications. Example, "I work out explanations on my own."
5. Implementation Activities: hands-on activities designed to increase exposure to STEM topics and real-world applications. Example, "We learn what scientists/technicians/engineers/ mathematicians or other STEM professionals do."

1 See Appendix A for information related to the construct reliabilities of the ALSQ.
Prepared by The Findings Group, LLC

Page 1

Executive Summary, continued

Results & Discussion

ALSQ Survey Constructs Table 2 summarizes students' responses to the ALSQ survey constructs across all programs. It is clear that the programs were effective at producing statistically significant increases in students' intrinsic motivation, self-management/self-regulation skills and intent to persist. The largest student gains observed were in the intrinsic motivation construct. Before the program, less than 53% of students indicated that they derive value and see the importance in learning about STEM; now, more than 76% say that they are intrinsically motivated to tackle STEM-related tasks and projects. Despite these statistically significant gains, it is important to note that the "now" scores across the following 3 constructs did not reach or exceed the optimal average of 4.0 on a 5-point Likert scale (1, strongly disagree to 5, strongly agree): Intent to Persist, Problem Solving, and Implementation Activities. See Figure 1. In order to maximize effectiveness, we would expect students' average scores to exceed 4.0. Figure 1 suggests that additional work may be needed in the above mentioned areas.

Table 2. Summary of Results by Constructs

Constructs

n

Before 920 Intrinsic Motivation
Now 912

Overall- Constructs

Mean1

Paired Samples
t-test

1 (Strongly Disagree)

2 (Disagree)

3 (Neutral)

4 (Agree)

5 (Strongly
Agree)

3.54 0.000** 4.10

+N+N+N+N+N+ +N+N+N+N+N+

5% 2%

10% 32% 31% 22% 4% 17% 34% 42%

Self- Management/Self-
Regulation
Intent to Persist

Before 919 Now 906 Before 915 Now 906

3.83 0.000** 4.11

+N+N+N+N+N+ +N+N+N+N+N+

19% 20%

12% 9%

25% 26% 18% 27%

18% 27%

3.38 0.000** 3.69

+N+N+N+N+N+ +N+N+N+N+N+

12% 10%

14% 10%

26% 17% 22% 19%

30% 39%

Problem Solving Now 913

3.93 N/A +N+N+N+N+N+ 3% 5% 22% 36% 34%

Implementation activities

Now 911

3.84 N/A +N+N+N+N+N+ 3% 6% 24% 36% 30%

1 Reference lines are set at 3.5 and 4. Negatively worded statements were reverse coded for mean computations. **p<.01, *p<.05, p<.10

5.00

Figure 1. Constructs

Before

Now

Optimal Average

4.50

**

4.00

**

**

3.50

3.00

2.50

2.00

1, Strongly Disagree to 5, Strongly Agree
Intrinsic Motivation Self-
Management/Self- Regulation
Intent to Persist Problem Solving Implementation
Activitites

**p<.01, *p<.05, p<.10; Scale is truncated for visual clarity. Prepared by The Findings Group, LLC

Constructs

Page 2

Executive Summary, continued
ALSQ Survey Constructs by Program Examining the ALSQ results by individual program, it is evident that all programs, with the exception of Murray STEM Academy, show statistically significant increases in intrinsic motivation, self- management/self-regulation and intent to persist. Students in the STEM for Life program at Carroll County and the Mechatronics program at Tift County show the largest increases from before to now on all three of the above mentioned constructs; Murray STEM Academy students show the smallest average increases across all 6 programs. See Table 3.

Table 3. Summary of Results by Constructs per Program

Overall- Constructs per Program

Constructs

STEM for Life Carroll County
(n=160)

Drew Charter
(n=273)

Murray STEM Academy
(n=37)

21st Century Barrow County
(n=318)

STEP Academy Moore MS
Sweetwater MS
(n=109)

TIFT County Mechatronics
(n=31)

Mean t-test Mean t-test Mean t-test Mean t-test Mean t-test Mean t-test

Intrinsic Motivation

Before Now

3.42 4.16

0.000**

3.75 4.16

0.000**

3.04 3.05

0.933

3.46 4.09

0.000**

3.41 4.12

0.000**

4.06 4.75

0.000**

Self-

Before 3.72

3.94

3.47

3.89

3.61

4.08

Management/ Self-Regulation

Now 4.22 0.000** 4.15 0.000** 3.55

0.512

4.09 0.000** 4.02 0.000** 4.50

0.000**

Intent to Persist

Before Now

3.22 3.64

0.000**

3.63 3.82

0.000**

2.83 2.84

0.908

3.29 3.62

0.000**

3.36 3.68

0.000**

3.79 4.59

0.000**

Problem Solving Now 3.92

3.99

3.00

3.99

3.77

4.60

Implementation activities

Now 3.81

n/a

3.90

n/a

2.91

n/a

3.93

n/a

3.72

n/a

4.57

n/a

Scale= 1, Strongly Disagree to 5, Strongly Agree. Negatively worded statements were reverse coded for mean computations. **p<.01, *p<.05, p<.10

In order for programs to maximize their effectiveness, we would expect "now" scores to reach or exceed the optimal average of 4.0. Figures 2 6 display "now" scores for each program and construct. For example, Figure 2 indicates that all programs met or exceeded the optimal average for intrinsic motivation, with the exception of Murray STEM Academy. In general, programs not reaching or exceeding the red horizontal line may need additional attention.

Figure 2. Intrinsic Motivation ("Now" Scores) 5

Figure 3. Self-Management/Self-Regulation ("Now" Scores)
5

4.5

4.5

4

4

3.5

3.5

3

3

2.5

2.5

Scale= 1, Strongly Disagree to 5, Strongly Agree. Scale was truncated for visual clarity.

STEM for Life Carroll County Drew Charter
Murray STEM Academy
21st Century Barrow County STEP Academy Moore and
Sweetwater MS TIFT County Mechatronics STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore and Sweetwater MS TIFT County Mechatronics

Prepared by The Findings Group, LLC

Page 3

Executive Summary, continued
Figure 4. Intent to Persist ("Now" Scores) 5 4.5 4 3.5 3 2.5

Figure 5. Problem Solving ("Now" Scores) 5 4.5 4 3.5 3 2.5

STEM for Life Carroll County
Drew Charter Murray STEM
Academy 21st Century Barrow
County STEP Academy Moore and Sweetwater MS
TIFT County Mechatronics STEM for Life Carroll County Drew Charter Murray STEM Academy 21st Century Barrow County STEP Academy Moore and Sweetwater MS TIFT County Mechatronics

STEM for Life Carroll County
Drew Charter Murray STEM
Academy 21st Century Barrow
County STEP Academy Moore and
Sweetwater MS TIFT County Mechatronics
STEM for Life Carroll County
Drew Charter Murray STEM Academy
21st Century Barrow County
STEP Academy Moore and Sweetwater MS TIFT County Mechatronics

Figure 6. Implementation Activities ("Now" Scores
5

Figure 7. Overall Ratings 5

4.5

4.5

4 4
3.5 3.5
3
3 2.5

2.5

2

Scale= 1, Strongly Disagree to 5, Strongly Agree. Scale was truncated for visual clarity. Program Rating Scale= 1, Very Poor to 5, Excellent.

Program Rating Collapsing across all programs, students' ratings exceeded the optimal average of 4.0. On a 5-point Likert scale where 1 signifies very poor and 5 signifies excellent, the average score was a 4.20. Looking at Figure 7, above, we see that 5 out of 6 programs were rated highly. However, Murray County STEM Academy may need additional assistance in improving student enjoyment.
Areas for Further Improvement Across all programs, further enhancing problem solving skills may be warranted. Specifically, students' ratings suggest that the inquiry-based learning environment may be improved by allowing students more opportunity to choose their own topics, work out explanations on their own, and plan and conduct their own projects. Likewise, encouraging programs to provide activities that foster interaction with STEM professionals may increase student exposure to real-world applications and careers. Such implementation activities may strengthen students' intentions and motivations to pursue additional education in STEM fields.

Prepared by The Findings Group, LLC

Page 4

Table 4. Intrinsic Motivation

Intrinsic Motivation

n

1. I prefer class work that is challenging so I can learn new things.

Before 920 Now 909

2. It is important to me to learn what is being taught in this program.

Before 919 Now 906

3. I like what I am learning in this program.

Before 913 Now 904

4. I think I will be able to use what I learn in this program in other classes.

Before 913 Now 897

5. Even when I do poorly on a test, Before 920

I try to learn from my mistakes. Now 912

6. I think that what I am learning in Before 906

this program is useful for me to

know.

Now 898

7. I think that what we are learning Before 911

in this program is interesting.

Now 907

8. Understanding STEM (Science, Technology, Engineering, and Math) is important to me.

Before 918 Now 907

9. I enjoy STEM (Science,

Before 915

Technology, Engineering, and

Math) in general.

Now 906

1 Reference lines are set at 3.5 and 4. **p<.01, *p<.05, p<.10

Mean1

Paired Samples
t-test

1 (Strongly Disagree)

2

3

(Disagree) (Neutral)

4 (Agree)

5 (Strongly Agree)

3.05

+N+N+N+N+N+ 11%

3.74 0.000** +N+N+N+N+N+

4%

17%

39% 23% 10%

6%

26% 40% 24%

3.83

+N+N+N+N+N+ 4%

4.33 0.000** +N+N+N+N+N+ 1%

5%

24% 39% 28%

2%

12% 34% 52%

3.55 4.12

0.000**

+N+N+N+N+N+ +N+N+N+N+N+

4% 2%

3.59

+N+N+N+N+N+ 4%

4.19 0.000** +N+N+N+N+N+

1%

9%

35% 33% 19%

3%

18% 35% 42%

10%

30% 36% 20%

4%

14% 36% 45%

3.94 4.40

0.000**

+N+N+N+N+N+ +N+N+N+N+N+

3% 1%

3.63

+N+N+N+N+N+ 4%

4.23 0.000** +N+N+N+N+N+

2%

6%

21% 35% 35%

2%

10% 30% 57%

9%

31% 33% 24%

3%

13% 33% 48%

3.41 3.97

0.000**

+N+N+N+N+N+ +N+N+N+N+N+

7% 3%

3.53

+N+N+N+N+N+ 6%

4.10 0.000** +N+N+N+N+N+

2%

11%

35% 31% 17%

5%

19% 38% 36%

10%

34% 25% 25%

4%

20% 32% 43%

3.31

+N+N+N+N+N+ 8%

3.87 0.000** +N+N+N+N+N+

4%

13%

36% 27% 16%

6%

23% 32% 34%

Prepared by The Findings Group, LLC

Page 5

Table 5. Self-Regulation/Self-Motivation

Self-Regulation/Self-Motivation

n

Mean1

Paired Samples
t-test

1 (Strongly Disagree)

2

3

(Disagree) (Neutral)

4 (Agree)

5 (Strongly Agree)

10. I turn all my assignments in on Before 919

time.

Now 904

3.48 3.87

0.000**

+N+N+N+N+N+ +N+N+N+N+N+

3% 2%

12%

34% 33% 17%

7%

23% 40% 29%

11. I miss class often. (n)

Before 911 Now 898

1.75 1.61

0.000**

+N+N+N+N+N+ +N+N+N+N+N+

60% 66%

19%

11%

7%

3%

17%

10%

4%

3%

12. I am often late for class. (n)

Before 900 Now 891

1.74 1.62

0.000**

+N+N+N+N+N+ +N+N+N+N+N+

57% 62%

22%

13%

5%

3%

21%

11%

4%

2%

13. I set aside time to do my homework and study.

Before 917 Now 903

3.30 3.66

0.000**

+N+N+N+N+N+ +N+N+N+N+N+

8% 6%

13%

34% 32% 13%

7%

26% 35% 26%

14. When I say I'm going to do something, I do it.

Before 914 Now 906

3.81 4.14

0.000**

+N+N+N+N+N+ +N+N+N+N+N+

2% 1%

5%

30% 34% 29%

3%

20% 34% 42%

15. I am a hard worker.

Before 914 Now 902

3.96 4.29

0.000**

+N+N+N+N+N+ +N+N+N+N+N+

2% 1%

5%

22% 37% 34%

3%

12% 34% 50%

16. I finish whatever I begin.

Before 911 Now 904

3.73 4.05

0.000**

+N+N+N+N+N+ +N+N+N+N+N+

2% 2%

1 Reference lines are set at 3.5 and 4. **p<.01, *p<.05, p<.10; (n) nega vely worded statement

8%

31% 33% 26%

3%

22% 36% 38%

Prepared by The Findings Group, LLC

Page 6

Table 6. Intent to Persist

Intent to Persist

n

17. I am considering a career in STEM (Science, Technology, Engineering, and Math).

Before 914 Now 905

18. I intend to get a college degree Before 915

in STEM (Science, Technology, Engineering, and Math).

Now 904

19. I can see myself working in STEM (Science, Technology, Engineering, and Math).

Before 913 Now 901

20. Someday, I would like to have a Before 913 career in STEM (Science,

Technology, Engineering, and Math).

Now 900

21. I intend to graduate from high Before 912

school

Now 906

1 Reference lines are set at 3.5 and 4. **p<.01, *p<.05, p<.10

Mean1

Paired Samples
t-test

1 (Strongly Disagree)

2

3

(Disagree) (Neutral)

4 (Agree)

5 (Strongly Agree)

3.03 0.000** +N+N+N+N+N+ 16%

18%

30% 18% 18%

3.39

+N+N+N+N+N+ 14%

11%

25% 21% 28%

3.15 0.000** +N+N+N+N+N+ 13%

3.49

+N+N+N+N+N+ 10%

16%

33% 20% 18%

10%

29% 22% 29%

3.03 0.000** +N+N+N+N+N+ 15%

3.39

+N+N+N+N+N+ 12%

17%

31% 21% 15%

12%

26% 27% 24%

2.99

+N+N+N+N+N+ 16%

0.000**

3.36

+N+N+N+N+N+ 11%

18%

33% 18% 15%

14%

27% 22% 25%

4.72 0.000** +N+N+N+N+N+ 1%

4.83

+N+N+N+N+N+ 1%

2%

4%

9%

84%

1%

2%

5%

91%

Prepared by The Findings Group, LLC

Page 7

Table 7. Problem Solving, Now Only

1

Problem Solving

n

Mean

Assessment

(Strongly

Disagree)

22. In this program, my teacher(s)

tells me how to improve my

913

4.17 Good +N+N+N+N+N+ 2%

work.

23. In this program, my teacher(s) lets us choose our own topics 908

3.39 Action ! +N+N+N+N+N+ 7%

or projects to investigate.

24. In this program, I work out explanations on my own.

911

3.67 Attention +N+N+N+N+N+ 2%

25. In this program, I have

opportunities to explain my

913

3.93 Attention +N+N+N+N+N+ 2%

ideas.

26. In this program, we plan and do

our own projects and/or

909

3.61 Attention +N+N+N+N+N+ 4%

experiments.

27. In this program, we work on real-world problems.

911

3.90 Attention +N+N+N+N+N+ 3%

28. In this program, we have class discussions.

913

4.19 Good +N+N+N+N+N+ 3%

29. In this program, we investigate to see if our ideas are right.

911

4.00 Good +N+N+N+N+N+ 2%

30. In this program, we need to be able to think and ask questions.

909

4.28 Good +N+N+N+N+N+ 2%

31. In this program, we are expected to understand and 913

4.17 Good +N+N+N+N+N+ 2%

explain ideas. 1 Reference lines are set at 3.5 and 4. Assessment: Good=Above 4.0; Attention=Below 4.0; Action=Below 3.5.

2 (Disagree)
2%
13% 5% 5%
11% 6% 3% 5% 2% 2%

3 (Neutral)
15%
33% 35% 23%
31% 23% 14% 20% 13% 18%

4 (Agree)
38%
27% 41% 40%
30% 36% 33% 39% 37% 36%

5 (Strongly
Agree) 42%
19% 17% 30%
25% 33% 47% 35% 48% 43%

Prepared by The Findings Group, LLC

Page 8

Table 8. Implementation Activities, Now Only

1

Implementation Activities

n

Mean

Assessment

(Strongly

Disagree)

32. In this program, my

teacher(s) takes notice of

911

3.89 Attention +N+N+N+N+N+ 4%

students' ideas.

33. In this program, my

teacher(s) shows us how new information relates to what

905

4.15 Good +N+N+N+N+N+ 2%

we have already learned.

34. In this program, we learn

what scientists/ technicians/ engineers/ mathematicians 909

3.77 Attention +N+N+N+N+N+ 4%

or other STEM professionals

do.

35. In this program, we do our work in groups.

907

3.71 Attention +N+N+N+N+N+ 3%

36. In this program, we interact

with scientists/ technicians/ engineers/ mathematicians

908

3.70 Attention +N+N+N+N+N+ 5%

or other STEM professionals. 1 Reference lines are set at 3.5 and 4. Assessment: Good=Above 4.0; Attention=Below 4.0; Action=Below 3.5.

2 (Disagree)
5% 3%
7% 6% 9%

3 (Neutral)
22% 16%
25% 33% 25%

4 (Agree)
38% 39%
36% 36% 32%

5 (Strongly Agree)
32%
41%
28%
23% 29%

Table 9. Educational Plans

What is the highest level of education you plan

Before

Now

Change1

to achieve?

n

%

n

%

n

%

High School

156 17%

53

6%

-103

-11.21%

2-year college

141 15%

118

13%

-23

-2.30%

4-year college

243 27%

186

21%

-57

-5.90%

Graduate School

185 20%

253

28%

+68

+7.97%

Professional School

185 20%

284

32%

+99

+11.44%

Total

910 100% 894

100%

Average2

2.91

3.35

0.000** (significant)3

1 Change from Before to Now. Increases are highlighted in green; decreases are highlighted in red.

2To compute averages, the following codes were applied: High School (1), 2-year college (2), 4-year college (3), Graduate School (4), Professional School (4). 3Paired

samples t-test, p-value: **p<.01, *p<.05, p<.10

Prepared by The Findings Group, LLC

Page 9

Table 10. Demographics
Gender
Female Male Total
Ethnicity
Asian Black Hispanic Native American White Multiracial Other Total

n 445 467 912

n

%

33

4%

327

36%

95

10%

4

0%

358

39%

75

8%

21

2%

913 100%

Grade
6th 7th 8th 9th 10th 11th 12th
Other
Total

% 49% 51% 100%
n 170 218 235 46 50 78 112
5 914

% 19% 24% 26% 5% 5% 9% 12% 1% 100%

Table 11. Participation

How long have you participated in this program?

n

0 semesters

4

1 semester

200

2 semesters

405

Dosage

3 semesters 4 or more semesters

40 143

Summer Only

1

Don't Know

120

Total

913

Did you participate in this program during the summer? n

No

806

Summer

Yes

50

Participation Don't Know

56

Total

912

% 0% 22% 44% 4% 16% 0% 13% 100%
% 88% 5% 6% 100%

Table 12. Program Rating

Program Rating: How would you n

rate this

program?

911

Mean1

Assessment

Very Poor (1)

Poor (2)

4.20 Good +N+N+N+N+N+ 3% 1%

1 Reference lines are set at 3.5 and 4. Assessment: Good=Above 4.0; Attention=Below 4.0; Action=Below 3.5.

Average (3)
15%

Good (4)
34%

Excellent (5)
47%

Prepared by The Findings Group, LLC

Page 10

Appendix A. Construct Reliabilities

Construct Reliabilities

Constructs

n

Intrinsic Motivation (9-items)

Before

801

Now

782

Before

807

Self-Management/Self-Regulation (7-items)

Now

784

Intent to Persist (5-items)

Before

828

Now

817

Problem Solving (10-items)

Now

816

Implementation Activities (5-items)

Now

821

Note. Construct reliabilities were computed based on December 2012 data.

Cronbach's alpha
.853 .863 .729 .708 .861 .877 .848 .756

Reliability Interpretation
Very good Very good
Good Good Very good Very good Very good Good

Cronbach's Alpha Reliability Key: Cronbach's alpha is a measure of the internal consistency of items in a construct. This statistic ranges from 0 to 1.00; the higher the value the better. An alpha of .80 or higher is considered to have achieved very good measurement reliability; an alpha of .65 is considered acceptable (Field, 2009). The table above suggests that all constructs achieved good to very good measurement reliability.

Reliability

Interpretation

.90 and above

Excellent reliability; at the level of the best measures

.80 - .90 Very good

.70 - .80 Good; in the range of most. There are probably a few items which could be improved.

Somewhat low. This measure needs to be supplemented by other measures (e.g., .60 - .70 more surveys) to determine outcomes. There are probably some items which could be
improved.

.50 - .60

Suggests need for revision of measure, unless it is quite short (ten or fewer items). The test definitely needs to be supplemented by other measures (e.g., more tests).

.50 or Questionable reliability. This measure should not contribute heavily to the outcomes below and needs revision.
From: J. C. Nunnally, Psychometric Theory. New York: McGraw-Hill, 1967, pp. 172-235.

Reference: Field, A. (2009). Discovering Statistics Using SPSS, 3rd Edition. Thousand Oaks, CA: Sage Publications.

Prepared by The Findings Group, LLC

Page 11