2015 Governor's Honors Program, evaulation report

2015 Governor's Honors Program, Evaluation Report
Nnenna A. Ogbu
December 2015

Executive Summary
OVERVIEW
The Georgia Governor's Honors Program (GHP) is a residential summer program for gifted and talented high school students who will be rising juniors and seniors during the program. The program offers instruction that is significantly different from the typical high school classroom and that is designed to provide students with academic, cultural, and social enrichment necessary to become the next generation of global critical thinkers, innovators, and leaders. In July 2013, Governor Nathan Deal signed an executive order transferring GHP from the Georgia Department of Education to the Governor's Office of Student Achievement (GOSA).1 In 2015, 680 high school students from 75 out of 189 Georgia counties attended GHP at the Valdosta State University campus. The 2015 cohort of GHP participants mark GOSA's second year in administering the program.
EVALUATION
The 2015 GHP Evaluation Report is an in-depth look at how GHP participants-- students, faculty/staff, and residential advisors--perceive the program and provides suggestions for further improvement. The Governor's Office of Student Achievement's (GOSA) Research and Evaluation unit conducted this evaluation. The evaluation team worked extensively with GHP administrators to identify the programmatic areas of most importance, design a plan to collect needed information, and analyze and report on the data. The 2015 GHP Evaluation Report includes:
A full demographic profile of students, faculty/staff, and residential advisors (where applicable),
A description of the evaluation instruments, An overview of the survey respondents as compared to the total
population, A presentation of the survey results by group, and A discussion of the findings, along with suggestions for future practice
ANALYSIS
The 2015 GHP Evaluation Report is divided into two main sections. The first section is a review of the program using administrative data. This section presents
1 To read the executive order, please click here: https://gov.georgia.gov/sites/gov.georgia.gov/files/related_files/document/07.26.13.02.pdf.
ii

demographic profiles for each segment of GHP participants. More demographic data is readily available for accepted students than for faculty/staff and residential advisors. Future evaluations would be strengthened by incorporating demographic data for all student applicants, not just those who were accepted, as well as basic demographic profiles for faculty/staff and residential advisors. Suggestions for basic demographic information include, but are not limited to: age, race/ethnicity, gender, education, experience, and occupation. Basic demographic data for the faculty/staff and residential advisors would allow for an evaluation of whether the faculty/staff and residential advisors are representative of the GHP student population and to determine whether survey respondents are representative of the full population.
The second section of the 2015 GHP Evaluation Report is a statistical analysis of the three online surveys given to each group of GHP participants. During the final week of the program, students, faculty/staff, and residential advisors were asked to complete an evaluation survey. Though the response rates vary per group--82% for students, 56% for faculty/staff, and 37% for residential advisors--each survey provides useful information about the experience of each group within GHP, as well as specific suggestions for future improvement.
MAJOR FINDINGS
Students Students were asked pre-post question pairs, questions used to retrospectively measure a student's expectation before arriving on campus to their perception at the end of the program. o About 97% of students expected to learn something new at GHP that would helpful to them later in life. At the end of the program, 92% of students still agreed. o About 90% of students expected to perform at a level higher than they usually do at their high school. Just over 83% of students still agreed at the end of the program. o Almost 95% of students expected to learn a lot from their GHP major compared to 88% of students that agreed that they learned a lot at the end of the program. o Only 87% of students expected to learn a lot from their GHP minor, and, at the end of the program, 84% of students agreed that they did learn a lot from their GHP minor. Ninety-seven percent of students would recommend GHP to a friend. This includes 96% of Academic majors and 99% of Performing Arts majors. Many students did not understand the necessity of the daily 7 PM hall checks, which were a time when all students were required to be in their dorms with their residential advisors. This was a common complaint amongst Performing Arts majors whose classrooms and class space were located on the other side of the Valdosta State University campus.
iii

Many students did not like the mandatory concerts. Students felt that the mandatory concerts limited the amount of time they had to study and practice, prioritized some GHP majors over others, and diminished the experience of the student performers and student patrons who would otherwise attend the concerts.
Faculty and Staff Many faculty and staff reported that their training time before students arrived on campus was best spent planning lessons as a department and networking with their colleagues, as opposed to attending a training session. Many of the GHP faculty and staff have taught at GHP for a number of years, though the exact figure is hard to estimate without basic demographic information for all faculty and staff. For those who previously taught at GHP before, the trainings provided little new information (self-report versus fact). Averages scores were high (4.8 on a 5-point scale) for the amount of support that instructors felt from their department chairs, and the usefulness of the observational feedback that instructors received from the Dean of Instruction. When asked about diversity, instructors suggest more racial diversity could be achieved within the faculty and staff body, and more gender and geographic diversity could be sought within the student body. Faculty and staff also urged the abolition of mandatory concerts.
Residential Advisors The chief concern for the residential advisors was the need for more upfront time to plan seminars, and more ongoing time during GHP to plan more seminars and additional activities. Residential advisors also expressed concern that the mandatory events and concerts for students cut into the amount of time they had with students to deliver seminars and encourage social activity.
iv

Table of Contents
EXECUTIVE SUMMARY ........................................................................................................... II OVERVIEW.................................................................................................................................... II EVALUATION ................................................................................................................................ II ANALYSIS ..................................................................................................................................... II MAJOR FINDINGS......................................................................................................................... III
TABLE OF CONTENTS ...............................................................................................................V TABLE OF FIGURES AND TABLES ....................................................................................... VI TABLE OF ABBREVIATIONS ................................................................................................VII INTRODUCTION ...........................................................................................................................8 DEMOGRAPHIC PROFILES.......................................................................................................9
STUDENTS ......................................................................................................................................9 FACULTY/STAFF AND RESIDENTIAL ADVISORS ...........................................................................16 SURVEY INSTRUMENTATION & DISSEMINATION .........................................................20 OVERVIEW ................................................................................................................................... 20 LIMITATIONS ...............................................................................................................................20 STUDENT EVALUATION SURVEY..................................................................................................21
Student Survey Responses.......................................................................................................22 Student Survey Analysis..........................................................................................................26
Pre-Post Question Pairs .................................................................................................................... 26 Means Testing .................................................................................................................................. 30 Open-Ended Responses .................................................................................................................... 33 Student Survey Summary ........................................................................................................34 FACULTY/STAFF EVALUATION SURVEY ......................................................................................36 Faculty/Staff Survey Responses ..............................................................................................37 Faculty/Staff Survey Analysis .................................................................................................39 Means Testing .................................................................................................................................. 40 Open-Ended Responses .................................................................................................................... 41 Faculty/Staff Survey Summary................................................................................................43 RESIDENTIAL ADVISOR EVALUATION SURVEY............................................................................44 RA Survey Responses..............................................................................................................45 RA Survey Analysis .................................................................................................................46 Means Testing .................................................................................................................................. 47 Open-Ended Responses .................................................................................................................... 49 RA Survey Summary ...............................................................................................................50 CONCLUSION ..............................................................................................................................51 APPENDIX A: 2015 GHP STUDENT EVALUATION SURVEY ITEMS..............................53 APPENDIX B: 2015 GHP FACULTY/STAFF EVALUATION SURVEY ITEMS................57 APPENDIX C: 2015 GHP RA EVALUATION SURVEY ITEMS...........................................59
v

Table of Figures and Tables Table 1: GHP Students by Race/ Ethnicity........................................................... 10 Table 2: GHP Students by Gender........................................................................ 10 Table 3: GHP Students by Race/Ethnicity and Gender ........................................ 11 Table 4: GHP Students by Current Grade Level .................................................. 11 Table 5: GHP Students by Metro Area or Private School .................................... 12 Table 6: GHP Majors by Area .............................................................................. 15 Table 7: GHP Majors by Area and Gender........................................................... 15 Table 8: GHP Faculty and Staff............................................................................ 17 Table 9: GHP Faculty and Staff by Instructional Area......................................... 18 Table 10: Residential Advisors by Gender ........................................................... 18 Table 11: Student Respondents by Gender ........................................................... 23 Table 12: Student Respondents by Current Grade Level...................................... 23 Table 13: Student Respondents by Race/Ethnicity ............................................... 24 Table 14: Student Respondents by GHP Major .................................................... 24 Table 15: Student Responses by VSUID Number ................................................ 25 Table 16: Duplicate Student Survey Responses by Gender.................................. 25 Table 17: Duplicate Student Survey Responses by Race/Ethnicity...................... 25 Table 18: Pre-Post Question Pairs ........................................................................ 26 Table 19: Faculty/Staff Respondents by Gender .................................................. 37 Table 20: Faculty/Staff Respondents by Race/Ethnicity ...................................... 38 Table 21: Faculty/Staff Respondents by Major .................................................... 38 Table 22: Faculty/Staff Respondents by Industry................................................. 39 Table 23: Faculty/Staff Respondents by Years Taught at GHP ........................... 39 Table 24: Faculty/Staff Average Scores for Scaled Responses ............................ 41 Table 25: RA Survey Response Rate.................................................................... 45 Table 26: RA Survey Respondents by Gender ..................................................... 45 Table 27: RA Respondents by Position ................................................................ 45 Table 28: RA Respondents by Occupation ........................................................... 46 Table 29: RA Respondents by Alumni Status ...................................................... 46 Table 30: RA Average Scores for Scaled Responses ........................................... 48 Table 31: 2015 GHP Student Evaluation Survey Items........................................ 53 Table 32: Scaled Response Average Scores for All Students............................... 55 Table 33: 2015 GHP Faculty/Staff Survey Items ................................................. 57
Figure 1: Map of Counties by Number of GHP Students, Statewide ................... 13 Figure 2: Map of GHP Students' Home High School, Metro-Atlanta Area......... 14 Figure 3: Student Response Rates......................................................................... 23 Figure 4: Student Survey, Rigor Pair 1 Post Question by Major.......................... 27 Figure 5: Student Survey, Rigor Pair 2 Pre Question by Geography ................... 28 Figure 6: Student Survey, Rigor Pair 2 Pre Question ........................................... 28 Figure 7: Student Survey, Courses Pair 1 Post Question by Major ...................... 29 Figure 8: Student Survey, Pre-Post Question Pairs Average Scores .................... 30
vi

Table of Abbreviations

GHP RA

Governor's Honors Program Residential Advisor

GOSA VSUID

Governor's Office of Student Achievement Valdosta State University Identification Number

vii

2015 GHP Evaluation Report
Introduction
The Georgia Governor's Honors Program (GHP) is a residential summer program for gifted and talented high school students who will be rising juniors and seniors during the program. The program offers instruction that is significantly different from the typical high school classroom and that is designed to provide students with academic, cultural, and social enrichment necessary to become the next generation of global critical thinkers, innovators, and leaders. In July 2013, Governor Nathan Deal signed an executive order transferring GHP from the Georgia Department of Education to the Governor's Office of Student Achievement (GOSA).2 The 2015 cohort of GHP participants mark GOSA's second year in administering the program.
GHP is held in mid-summer (mid-June to mid-July) as a residential educational experience on a college or university campus. Students attend classes in the mornings and afternoons in specific areas of study, and they participate in a wide variety of social and instructional opportunities every evening. Meals and rooms are provided by the program with the only required and expected costs to the students being travel expenses to and from the interviews, a few basic supplies for classes and dorm rooms, and spending money as desired during the program.3
The 2015 GHP Evaluation Report is an in-depth look at how GHP participants-- students, faculty/staff, and residential advisors--perceive the program and provides suggestions for further improvement. The Governor's Office of Student Achievement's (GOSA) Research and Evaluation unit conducted this evaluation. The evaluation team worked extensively with GHP administrators to identify the programmatic areas of most importance, design a plan to collect needed information, and analyze and report on the data. The 2015 GHP Evaluation Report includes:
A full demographic profile of students, faculty/staff, and residential advisors (where applicable)
A description of the evaluation instruments An overview of the survey respondents as compared to the total
population A Presentation of the survey results by group And a discussion of the findings, along with suggestions for future
practice
2 To read the executive order, please click here: https://gov.georgia.gov/sites/gov.georgia.gov/files/related_files/document/07.26.13.02.pdf. 3 For more information about the Governor's Honors Program, please visit the program's website: https://gosa.georgia.gov/governors-honors-program.
8

2015 GHP Evaluation Report
Demographic Profiles STUDENTS
The Governor's Honors Program (GHP) accepted 682 students from all over Georgia into its 2015 GHP Student cohort. This report is limited to only students who were accepted into GHP. Student demographic information was collected when students applied for admission and when students arrived on campus for the program. Demographic information includes a student's race/ethnicity, gender, current grade level, nomination area, and GHP major. Tables 1-5 provide an overview of the student demographics. Race/Ethnicity In the application, students were asked to select their race/ethnicity from a list of categories. Table 1 shows the distribution of GHP students by race/ethnicity. The 2015 GHP student population was predominately White (54%) and Asian (22%). The next largest categories of students were African American (8%), Latino (4%), Multi-Racial (4%), and Native American/Alaskan Native (<1%). About 6% of students preferred not to answer, and the remaining 1.5% of students chose Other or did not supply an answer. Table 1 also includes a comparison of the GHP student population to the statewide population of public school students. GHP has a much larger percentage of Asian students, and much smaller percentages of African American and Latino students, than exist proportionally in the state.
9

2015 GHP Evaluation Report

Table 1: GHP Students by Race/ Ethnicity

Race/Ethnicity

Total GHP

Number

White, non Hispanic

368

Asian/ Pacific Islander

151

African American/ Black

54

Latino/ Hispanic

28

Multi-Racial

28

Native American/ Alaskan Native

2

Other

4

Prefer not to answer

44

No Response

3

Total

682

1Source: GOSA Report Card, 20154

Total GHP Percent 54%
22%
8%
4% 4%
<1%
<1% 6% <1% 100%

Percent in Georgia1
43%
3%
37%
13% 3%
<1%
N/A N/A N/A 100%

Gender

Students were asked to select their gender, which was subsequently used to assign on-campus housing. Table 2 shows that, overall, 45% and 55% of the 2015 GHP student population was male and female, respectively. A cross tabulation of data shows that the number of female students matched or outnumbered the number of male students in each racial category, with the exception of Latino students and students who preferred not to answer. Table 3 is the distribution of GHP students by race/ethnicity and gender. The Latino population is the only student group in which male students outnumber female students.

Table 2: GHP Students by Gender

Gender

Total Number

Female

376

Male

306

Total

682

Total Percent 55% 45% 100%

4 To view full annual reports from the GOSA Report Card, please see the GOSA website: gosa.georgia.gov\report-card.
10

2015 GHP Evaluation Report

Table 3: GHP Students by Race/Ethnicity and Gender

Female Male Total

White, non Hispanic

190

178

368

Asian/Pacific Islander

93

58

151

Latino/Hispanic

10

18

28

Multi-Racial

19

9

28

Native American/ Alaska

1

Native

Other

2

1

2

2

4

African American/ Black

32

22

54

No Response

2

1

3

Prefer not to answer Total

27

17

44

376

306

682

Grade
GHP is open to all 10th and 11th graders who attend high school in Georgia. This year, the program accepted about 2.5 times more 11th graders than 10th graders, putting the total populations at approximately 72% 11th graders and 28% 10th graders (see Table 4).

Table 4: GHP Students by Current Grade Level

Total Number Total Percent

10th grade

193

28%

11th grade

488

72%

Total

681

100%

Geography

GHP accepts students from all over the state to attend the program. GHP students attend high school in over 75 of Georgia's 189 counties, including city school systems and private schools. Although GHP students attend high schools throughout the state, the majority of 2015 GHP students attend high school in the metro Atlanta area. Metro Atlanta includes Cherokee, Clayton, Cobb, DeKalb, Douglas, Fayette, Fulton, Gwinnett, Henry, and Rockdale counties.5 Table 5 shows that students from the metro Atlanta counties make up about 59% of the 2015 GHP student population. Private school students from around the state make up another 6%, and the remaining 35% of students came from areas outside of metro Atlanta. According to the 2014 U.S. Census, about 43% of 14-to-17 year olds live in Metro

5Here, "metro" is defined as the ten counties included in the Atlanta Regional Commission's regional planning and intergovernmental coordination.
11

2015 GHP Evaluation Report

Atlanta counties, and the other 57% live in non-metro Atlanta counties.6 Figure 1 is a map of counties by the number of GHP students who attend high schools in that county. This figure includes all GHP students from public and private schools, county and city school systems, and metro Atlanta and non-metro Atlanta schools.

Table 5: GHP Students by Metro Area or Private School

Total GHP Total GHP Total Statewide

Students Percentage

Percentage

Metro Atlanta

402

59%

43%

Non-metro Atlanta

240

35%

57%

Private School

40

6%

N/A

Total

682

100%

100%

6 Statewide enrollment percentages for 10th and 11th grade students comes from the 2014 U.S. Census Bureau's American FactFinder profile for Georgia.
12

2015 GHP Evaluation Report
Figure 1: Map of Counties by Number of GHP Students, Statewide
Figure 2 is a closer look at the high schools within metro Atlanta. This map separates students by public school, which includes city school systems, and private school. Though 59% of GHP students attend metro Atlanta schools, the map shows that the majority of those schools and students are located in north metro Atlanta.
13

2015 GHP Evaluation Report
Figure 2: Map of GHP Students' Home High School, Metro Atlanta Area
GHP Academics GHP offers twelve majors that are characterized as either Academic Majors or Performing Arts Majors (see Table 6). Table 7 shows the distribution of GHP majors by gender. Music was the largest GHP major (16%), followed by Communicative Arts (12%), Mathematics (12%), Science (12%), Social Studies (12%), and World Languages (11%). The remaining majors each accounted for less than 10% of the total 2015 GHP student population. Though female students outnumber male students in the overall GHP student population, the proportion of female students to male students does not carry through each of the 12 GHP majors. For example, female students outnumber male students 7:1 in Dance, nearly 5:1 in Visual Arts, and 3:1 in Communicative Arts and World Languages. Male students outnumber female students 4:1 in Engineering, and nearly 2:1 in Mathematics.
14

2015 GHP Evaluation Report

Table 6: GHP Majors by Area Academic Majors Agricultural Science Communicative Arts Engineering Mathematics Science Social Studies World Languages:
Chinese French German Latin Spanish

Performing Arts Majors
Dance Music:
Brass Jazz Percussion Piano Strings Voice Woodwinds Theatre Performance Theatre Tech Visual Arts

Table 7: GHP Majors by Area and Gender

Female Students

Number Percent

Agricultural Science

16

59%

Communicative Arts

60

75%

Dance

22

88%

Engineering

9

45%

Mathematics

30

38%

Music

45

41%

Science

39

49%

Social Studies

37

45%

Technology

5

20%

Theatre

16

52%

Visual Arts

38

83%

World Languages

58

76%

Total

375

Male Students Number Percent

11

41%

20

25%

3

12%

11

55%

50

63%

64

59%

41

51%

45

55%

20

80%

15

48%

8

17%

18

24%

306

Total Students Number Percent

27

4%

80

12%

25

4%

20

3%

80

12%

109

16%

80

12%

82

12%

25

4%

31

5%

46

7%

76

11%

681

100%

Based on the student demographic data, GHP should seek to broaden student diversity in a few key areas. First, racial diversity would broaden by increasing the number of African American and Latino students that attend GHP. In 2015, African Americans comprised 8% of GHP's student population and Latinos comprised 4%.

15

2015 GHP Evaluation Report
Statewide, however, African Americans and Latinos account for 37% and 13%, respectively, of Georgia's public school students. Recruitment efforts could focus on encouraging nominations from schools serving larger populations of gifted and talented African American and Latino students.
Gender diversity can also be broadened within GHP majors. For example, female students make up 20% of the population of GHP Technology majors; male students make up only 12% and 17% of GHP Dance and Visual Arts majors, respectively. Recruitment efforts for future GHP cohorts could also focus of finding underrepresented gifted and talented students who would do well in these GHP majors.
Geographic diversity is the final area that GHP could look to broaden diversity. Fifty-nine percent of GHP students attend high school in metro Atlanta. This is not dissimilar to the total statewide population of students attending high school in metro Atlanta. However, a closer examination of the metro Atlanta map shows that most of those students attend schools in north metro Atlanta, above Interstate 20. GHP could broaden geographic diversity by targeting students in the south metro Atlanta area and encouraging nominations from schools in the area with gifted and talented students.
Limitations
GHP collects and maintains a thorough amount of student demographic data for all of its accepted finalists. However, though demographic information is collected for all student applicants, the information is not stored for students who were not accepted. The archived demographic data could include basic information on a student's race, gender, grade, and high school, as well as information on a student's nomination area, nominator, application scores, and interview scores. If the demographic information for all student applicants were available, this evaluation could expand the analysis of GHP students between those who were accepted into the program and those who were not. Given the full population of applicants, the evaluation could highlight trends within the accepted finalists and report on descriptive information, such as the acceptance rate. Both types of reported information could be useful to the program as it develops information guides for future students and nominators.
FACULTY/STAFF AND RESIDENTIAL ADVISORS
Faculty and Staff
There were 57 instructional faculty and staff, which includes all personnel except for the Residential Life Advisors and the program administrators (see Table 8). In Table 9, faculty and staff are described by their instructional area or area of
16

2015 GHP Evaluation Report

expertise. For faculty who taught a GHP major, the instructional area is the same as their GHP major. For staff who taught a GHP elective or supported students in a specific role, the instructional area is their area of expertise. Several faculty taught both majors and electives, some staff taught only electives, and other staff supported students by performing a specific role.

Table 8: GHP Faculty and Staff Total Number

Faculty

51

Staff

Counselors

4

Library/Media

1

Whitehead Auditorium/Control

1

Total

57

Total Percentage 89%
7% 2% 2% 100%

Specific information on each faculty or staff member's gender is not contained in GHP's administrative data. Unlike the students and the RAs who were either assigned to one dorm for males or one dorm for females, GHP faculty and staff who chose to live on-campus were assigned to one co-educational dorm. For the students and the RAs, one can infer the individual gender based on the individual's dorm assignment. The same inference cannot be made for the total population of GHP faculty and staff because both males and females were housed in the same dorm.

17

2015 GHP Evaluation Report

Table 9: GHP Faculty and Staff by Instructional Area

Total Number of Total Percentage

Instructors

of Instructors

Agricultural Science

2

4%

Communicative Arts

5

9%

Computer Tech

2

4%

Counselor

4

7%

Dance

2

4%

Design & Engineering

2

4%

Engineering Technology

1

2%

Fitness

2

4%

Lead Computer Tech

1

2%

Mathematics

5

9%

Media Specialist

1

2%

Music

9

16%

Science

5

9%

Social Studies

5

9%

Theatre

2

4%

Theatre Design

1

2%

Visual Arts

3

5%

World Languages

5

9%

Totals

57

100%

Residential Advisors

There were 34 Residential Advisors (RAs), which includes two Dorm Directors, who helped to supervise students and create structured activities outside of instructional time. Table 10provides a quick breakdown of RA titles. Similar to a traditional college campus environment, the RAs lived on campus amongst the students. The RAs were responsible for creating and hosting the "mixed sessions" activities which took place between 4:30 PM 7:30 PM, after students were finished with classes for the day.

Table 10: Residential Advisors by Gender

Male

Female

Residential Advisors Dorm Directors Total

14 1 15 (44%)

18 1 19 (56%)

Total 32 2 34

18

2015 GHP Evaluation Report
Limitations Data on the demographic profiles of the Faculty/Staff and RAs are limited. Unlike the student group, in which basic demographic information is collected administratively during the application and documentation stages, Faculty/Staff and RAs are not required to complete or submit demographic profiles. Suggestions for practice include collecting and archiving basic demographic data during the application stage for Faculty/Staff and RAs. Basic demographic data could include, but is not limited to: age, race/ethnicity, gender, education, experience, and occupation. Basic demographic data for the Faculty/Staff and RAs would allow for an evaluation of whether the faculty/staff and RAs are representative of the GHP student population. Additionally, demographic records could also help to answer questions related to changes in the GHP Faculty/Staff and RAs over time. For example, years of experience with GHP, occupation types outside of GHP, and attrition rates could all be examined for Faculty/Staff and RAs.
19

2015 GHP Evaluation Report
Survey Instrumentation & Dissemination
OVERVIEW
For the 2015 GHP evaluation, three online surveys were designed and disseminated to the three groups of GHP participants: students, faculty/staff, and RAs. The evaluation team used a combination of GHP programmatic goals and GHP administrator priority areas to identify the most important constructs and the measurement requirements. For example, one targeted outcome for student satisfaction states that "95% of the students agree or strongly agree that the Residence Life program was well organized and assisted them in enjoying GHP." The percentage (95%) and language (agree or strongly agree) indicated that a Likert scale assessment item would be most suitable for collecting feedback. Each survey had its own content and priority areas, and the groups had various amounts of time and reminders to complete the online surveys.
LIMITATIONS
The chosen methods for the 2015 GHP evaluation have limitations. First, the evaluation surveys are confidential, but not anonymous by design. Each survey response requires a valid Valdosta State University ID (VSUID) because it was important to the evaluation team and GHP administrators to be able to match survey data to GHP participants. Data was matched by VSUID, which is the only verifiable and uniformly-assigned identifier for each GHP participant. Matching allowed the evaluation team to verify that each survey response was submitted by a current GHP participant. Matching also made more data available for comparisons between survey respondents and non-respondents without asking respondents to submit duplicate data.
Second, to allow for ease in survey administration and analysis, the evaluation surveys only available online, and the survey links were only posted in email messages. Paper-and-pencil surveys were not feasible because they would not allow for uniform collection, data authentication, information verification, or data conversion of hand-written data into electronic data. The evaluation survey links were also not posted on the central GHP website. The GHP website is used to communicate information to the broad public. Posting a link to an evaluation survey on the public site may have encouraged invalid submissions.
Third, the evaluation surveys were sent out during the final week of GHP. According to GHP administrators, the final week of the program tends to be the most time-consuming and laborious week of the program. Students are finalizing and presenting final projects and performances. Instructors are working to evaluate final assignments, projects, and performances in time to give students feedback
20

2015 GHP Evaluation Report
before GHP concludes. RAs are working to keep students safe and on track as an influx of outside visitors arrive on campus for final performances and packing, and then to reset their residence hall after students depart campus. During the final week of the program, GHP participants--students, faculty/staff, and RAs--have many important activities vying for their attention. The response rates for the surveys may have been higher if the surveys were sent out in the third week. However, the participants' 2015 GHP experience would still be ongoing and their opinions subject to change.
Finally, all of the data from the evaluation surveys is self-reported. Though each group of participants received emails from the GHP program manager to complete the evaluation survey, each group may have been subtly reminded by other authority figures and subsequently influenced in different ways. For example, in a coordinated effort to encourage student responses, GHP students received in-person reminders from their RAs, every day at 10:30 PM for four days. These daily reminders may have been what influenced the number of references, both positive and negative, to RAs in surveys that were started or submitted between 10:3011:00 PM. A similar coordinated effort was not strategized for faculty/staff or RA responses, but the two groups may have received additional reminders to complete the survey from other authority figures (i.e., Deans or department chairs). Currently, however, this evaluation does not attempt to make any such causal links.
STUDENT EVALUATION SURVEY
The 2015 GHP Student Evaluation Survey was the longest of the three surveys, and consisted of 45 total questions. The questions were organized by topic area, which included, in order of appearance: general information (3), program rigor (9), residence life (6), courses (6), instructors (6), future plans (6), other (7), and demographics (3). Except for the first question, which instructed respondents to enter their 9-digit VSUID number, all of the survey items were optional. Thirtyeight of the survey items were five-point Likert scale questions, four of the items were drop-down selections, three items were open response, and one item was for the VSUID numbers.
Three of the topic areas--rigor, instructors, and residence life--were especially important to GHP given its entirely special population of gifted and talented high school students and its program delivery model. It is important for GHP to offer rigorous coursework and advanced-level content because the program targets the state's gifted and talented student population. In their normal high schools, gifted and talented students complete rigorous and advanced coursework and are also frequently amongst the school's highest-achieving students. Much of the 2015 GHP student evaluation survey asked students for their expectations and perceptions of GHP coursework and content in relation to students' normal high school coursework and content. The GHP delivery model is also unique for high school
21

2015 GHP Evaluation Report
students because attending GHP requires students to live, supervised, on a college campus for four weeks during the summer. The intensity and duration of the residency is novel for many high school students, and one of GHP's top priorities is to provide students with a safe and supportive living environment.
The student evaluation survey was sent to students via email during the final week of GHP. Students received one initial invitation email to complete the student evaluation survey, and one reminder email to complete the student evaluation survey. The evaluation team also coordinated with GHP administrators to use the daily 10:30 PM hall checks as a time for students to receive in-person survey reminders from their RAs and complete the survey using their own devices. Though these in-person reminders are not verifiable, they were scheduled to occur every night for four nights. The student evaluation survey remained accessible for a total of seven days.
Student Survey Responses
The 2015 GHP student evaluation survey was sent to 669 GHP students. Although GHP accepted 682 students for the 2015 cohort, thirteen students (<2%) either did not attend or did not complete the program. Out of the remaining 670 students, one student did not have a valid email address to be included in the initial survey invitation.
Tables 11-14 show that, in general, the student survey respondents closely mirror the total 2015 GHP student population by gender, grade level, race/ethnicity, and major. Figure 3 shows that a total of 557 students responded to the survey, which is a response rate of about 82%. Each set of survey question types received between 520-530 responses, with the exception of questions related to future plans which received between 480-500 responses. This is the highest response rate of the three evaluation surveys. Table 11 shows that, of the students who responded, 306 (55%) were female and 251 (45%) were male. Table 12 is a summary of student survey respondents by grade and compared to the total GHP student population. Table 13 is a table of student respondents by race/ethnicity. The majority of respondents were White students (53%) and Asian students (23%) who, similar to the total population, constituted over 75% of all survey respondents. African American students were 7% of respondents, followed by those preferring not to answer (7%), Multi-Racial (4%), Latino (4%), and other categories each less than 1%. The majority of survey respondents were Music majors (16%), followed by Science majors (13%), and then Math, Social Studies, and World Languages majors each contributing 12% (see Table 14).
22

2015 GHP Evaluation Report

Figure 3: Student Response Rates
Survey Responses
125 18%

Response No Response

557 82%

Table 11: Student Respondents by Gender Number of Percentage of Percentage of
Respondents Respondents GHP Students

Male

251

45%

45%

Female

306

55%

55%

Total

557

100%

100%

Table 12: Student Respondents by Current Grade Level Number of Percentage of Percentage of
Respondents Respondents GHP Students

10th grade

160

29%

28%

11th grade

396

71%

71%

Not Reported

1

<1%

<1%

Total

557

100%

100%

23

2015 GHP Evaluation Report

Table 13: Student Respondents by Race/Ethnicity Number of Percentage of
Respondents Respondents

White, non Hispanic

297

53%

Percentage of GHP Students
54%

Asian/Pacific Islander

127

23%

22%

African American/Black

41

7%

8%

Latino/Hispanic

24

4%

4%

Multi-Racial Native American/Alaskan Native Other

25

4%

4%

2

<1%

<1%

3

<1%

<1%

Prefer not to answer

37

7%

6%

Not Reported

1

<1%

<1%

Total

557

100%

100%

Table 14: Student Respondents by GHP Major Number of Percentage of
Respondents Respondents

Agricultural Science

23

4%

Communicative Arts

62

11%

Dance

21

4%

Engineering

15

3%

Mathematics

65

12%

Music

88

16%

Science

70

13%

Social Studies

68

12%

Technology

23

4%

Theatre

19

3%

Visual Arts

36

6%

World Languages

66

12%

Not Reported

1

<1%

Total

557

100%

Percentage of GHP Students
4% 12%
4% 3% 12% 16% 12% 12% 4% 5% 7% 11%
-100%

24

2015 GHP Evaluation Report

A number of students accessed the survey multiple times, and either completed or partially completed the survey multiple times as well. These 88 duplicate responses have not been dropped from the analysis because, in many instances, the scaled answers and the open responses vary from the initial response. Table 15provides detail regarding how many duplicate responses were received. Table 16 andTable 17 provide greater detail about duplicate responses by gender and race/ethnicity. More than half of the duplicate responses came from female students. The majority of duplicate responses were submitted by White students (54%) and Asian students (29%).

Table 15: Student Responses by VSUID Number

Number of

Number of

Percentage of

Survey Entries Respondents

Total Responses

1

557

86%

2

73

11%

3

12

2%

4

2

<1%

5

1

<1%

Total

645

100%

Table 16: Duplicate Student Survey Responses by Gender

Number of

Percentage of

Duplicate Responses Duplicate Responses

Female

51

58%

Male

37

42%

Total

88

100%

Table 17: Duplicate Student Survey Responses by Race/Ethnicity

Number of

Percentage of

Duplicate Responses Duplicate Responses

White, non Hispanic

47

54%

Asian/Pacific Islander

26

29%

Latino/Hispanic

3

3%

Multi-Racial

3

3%

Native American/Alaskan Native

1

1%

Other

1

1%

African American/Black

4

5%

Prefer not to answer

3

3%

Total

88

100%

25

2015 GHP Evaluation Report

Student Survey Analysis

The analyses focus on differences between average scores by student group, namely by major area, by gender, by specific major, by race, and by metro/private. The analyses were conducted using a one-way analysis of variance (ANOVA) test by group. A one-way ANOVA compares the average score for each category within a group--for example, Academic majors versus Performing Arts majors or males versus females--to see if the average score for each category within a group is statistically different than the average score for another category within the same group.

The following analyses will focus, first, on the pre-post question pairs within the student evaluation survey. For this section, in addition to a one-way ANOVA between student groups, paired sample t-tests are also performed to test for statistical differences in the total average scores of each pre-post question pair. Second, the categorical averages per group will be analyzed for all survey items. Discoveries from the open-ended student comments will be presented last.
Pre-Post Question Pairs

Pre-post question pairs were used to retrospectively measure students' expectations before arriving on campus for GHP. Table 18 shows each of the pairs and specific questions within each pair. The "pre" question is designed to provide a base level for student expectations, and the "post" question is designed to indicate if student expectations were met. One-way ANOVA testing was used to compare group means for the student survey's pre-post question pairs. For each question, only groups whose categories are statistically different are presented below. The difference in average scores between categories does not need to vary by a large number to be deemed statistically significant. Statistical significant is more about the pattern of responses from individuals within a group and if that pattern is uniquely different based on whether the individual is, for example, male or female. Average scores for each a category are presented in parentheses.

Table 18: Pre-Post Question Pairs PRE Question
Rigor Pair 1 I expected to learn something new at GHP that would be helpful to me later in life.
Rigor Pair 2 I expected to perform at a level higher than I usually do at my high school.
Courses Pair 1 I expected to learn a lot from my GHP Major.
Courses Pair 2 I expected to learn a lot from my GHP Elective.

POST Question I learned something new at GHP that will be helpful to me later in life. At GHP, I performed at a level higher than I usually do at my high school. I learned a lot from my GHP Major. I learned a lot from my GHP Elective.

26

2015 GHP Evaluation Report

For the rigor pair 1 post question, which asked students if they learned something new at GHP that will be helpful later in life, nearly 97% of students agreed, though, certain majors differed in how much they agree with the post question. Figure 4 shows the averages for each major. Majors with a patterned column differed significantly from each other. Compared to Communicative Arts majors, Agricultural Science, Science, and World Languages majors all held significantly lower perceptions about what they learned at GHP. The remaining GHP major categories did not display any significant differences.
Figure 4: Student Survey, Rigor Pair 1 Post Question by Major

Rigor Pair 1 Post Question, by Major

5.0

4.4

4.9

4.3

4.6

4.5

4.0

4.2

4.6

4.9

5.0

4.6

4.3

4.0

3.0

2.0

1.0

1=Strongly Disagree; 5=Strongly Agree
For the rigor pair 2 pre question, which asks if students expected to perform at a level higher than they usually do, 92% of students agreed. Though, performing arts majors (4.7) held higher expectations than academic majors (4.5). Additionally, Figure 5shows that private school students held significantly higher expectations than did public school students, both metro and non-metro, regarding their level of performance at GHP.

27

2015 GHP Evaluation Report

Figure 5: Student Survey, Rigor Pair 2 Pre Question by Geography

Rigor Pair 2 Pre Question, by Geography

5.0 4.5

4.7

4.2

4.0

3.0

2.0

1.0 Metro Atlanta

Non-metro Atlanta

Private School

1=Strongly Disagree; 5=Strongly Agree

Figure 6: Student Survey, Rigor Pair 2 Post Question

5.0 4.3 4.5 4.0 3.0 2.0 1.0

Rigor Pair 2 Post Question

5.0

4.5 4.1 4.4 4.3

4.7 4.2

4.3 4.4 4.0

Major Area Race Geography

1=Strongly Disagree; 5=Strongly Agree
The rigor pair 2 post question asked students about their perceptions of the level of performance they had to achieve at GHP compared to their regular high school. About 83% of students agreed that they had to achieve at a higher level at GHP. Figure 6shows the relationships by major area, race/ethnicity, and geography. Performing Arts majors (4.5) held significantly higher perceptions than Academic
28

2015 GHP Evaluation Report

majors (4.3). Asian students (4.1) held significantly lower perceptions about their level of performance at GHP than did White students (4.5). Compared to public school students outside of Metro Atlanta (4.4), private school students held significantly lower perceptions about their level of performance at GHP (4.0).
Figure 7: Student Survey, Courses Pair 1 Post Question by Major

Courses Pair 1 Post Question, by Major

4.9

4.9 5.0

5.0 4.4 4.0

4.5 4.5

4.6

3.9

4.1 3.6

4.5 4.1

3.0

2.0

1.0

1=Strongly Disagree; 5=Strongly Agree
When it comes to the courses pair 1 post question, the perception that a student learned a lot from his or her GHP major, almost 95% of students agree that they did learn a lot from the GHP major. Communicative Arts majors had higher perceptions than Dance, Agricultural Science, Science, and World Languages majors. Additionally, Theatre Performance majors held higher perceptions of how much they learned from their GHP major than did Agricultural Science and Social Studies majors. Figure 7 shows the differences between the two sets of majors using two different styles of patterned columns. No significant differences were found between the courses pair 1 pre question or either of the courses pair 2 pre-post questions.
Though between 83% and 97% of all students agree with the pre-post questions pairs, all of the pairs appeared to have lower average agreement for the post question. To test this hypothesis, paired sample t-tests were conducted between each pair's overall average. Rigor pair 1, rigor pair 2, and courses pair 1 all have statistically lower averages for the "pre" question than they do for the "post" question. Figure 8 denotes these statistical differences with an asterisk. The lower averages for the "post" questions could suggest that student expectation were not met in regards to how much they would learn at GHP, how hard they would have to work at GHP, and how much they learned from their GHP major. Given the scaled nature of the data, a look at the pre-post questions reveals that the differences
29

2015 GHP Evaluation Report

are in the number of students selecting "5=Strongly Agree" in the pre question, and "4=Agree" or "3=No Opinion" in the post question.
Figure 8: Student Survey, Pre-Post Question Pairs Average Scores

Average Scores for Pre-Post Question Pairs

5.0 4.7 4.5

4.6 4.3

4.7 4.4

4.4 4.3

4.0

3.0

2.0

1.0

Rigor Pair 1*

Rigor Pair 2*

Courses Pair 1*

Courses Pair 2

Pre Question Post Question

1=Strongly Disagree; 5=Strongly Agree

Means Testing
In addition to the pre-post question pairs, ANOVA testing was used to compare group means for all five-point Likert scale survey items on the student evaluation survey. The statistically significant results are presented by student group: grade, gender, major area, GHP major, and race. A full table of the overall average scores for each five-point Likert scale survey item is available in Appendix A.
Differences by Grade & Gender When asked if they already possessed all the necessary skills to succeed in college, 11th graders (3.6) displayed higher levels of agreement than 10th graders (3.2). Similarly, male students (3.7) tended to agree more than female students (3.4) when asked the same question. Male students and female students differ slightly on five questions. Male students (4.7) agreed more than female students (4.5) when asked

30

2015 GHP Evaluation Report
if they felt that the GHP instructors helped them to be successful. Female students (3.4) tended to agree more than their male counterparts (3.2) that the counseling services were useful. Female students (4.0) also tended to agree more than male students (3.7) that GHP students were tolerant of diverse opinions. Finally, when asked if GHP helped them to learn more about careers in their interest areas, male students (4.1) tended to agree more than female students (3.8).
Differences by Major Area Performing arts students tended to have higher levels of agreement than academic students. In addition to the pre-post paired questions regarding expectations and perceptions of performance level, performing arts students (4.7) had higher levels of agreement than academic majors (4.5) when asked if GHP was academically/artistically challenging and if GHP coursework is more rigorous than what they normally experience (4.2 versus 3.6). Performing arts students (4.7) agreed more than academic students (4.4) when asked if they almost always use their own device(s) to connect to the internet.7 However, academic students (4.6) agreed slightly more than performing arts students (4.5) that their GHP elective offered a unique learning experience. Academic students (4.7) also agreed more than performing arts students (4.5) when asked if the GHP instructors were tolerant of diverse opinions. Lastly, academic students (3.6) agreed more than performing arts students (3.3) when asked if they already had all of the necessary skills to succeed in college before coming to GHP.
Differences by GHP Major Several significant differences existed between majors when asked about program rigor. Compared to Music majors (4.4), Communicative Arts (4.9), Mathematics (4.9), and Theatre Performance (4.9) majors all held higher levels of agreement when asked if they learned something unexpected at GHP. Engineering (4.2), Agricultural Science (3.4), and World Languages (4.2) majors all agreed less than Music majors (4.7) that GHP was academically or artistically challenging. Dance (4.1), Engineering (4.4), Science (4.4), Social Studies (4.4), and World Languages (4.4) majors all agreed less than Music majors (4.7) that GHP was socially engaging. Several majors--Engineering (3.3), Agricultural Science (3.3), Science (3.4), Social Studies (3.5), and World Languages (3.3)--agreed less than Music majors (4.2) that the GHP coursework was more rigorous than what they are accustomed to at their regular high school. Finally, Agricultural Science (4.3) and Social Studies (4.5) majors agreed less than Music majors (4.6) that they are satisfied with GHP.
A few differences existed between majors when students were asked about Residential Life activities. Music majors (4.3) tended to agree more than Social Studies (3.9) and Theatre Performance (3.4) majors that the Residential Life seminars and activities improved their GHP experience. Social Studies majors (4.3)
7 The survey item regarding personal technology was a direct request by GHP administrators in the hopes of gathering information regarding students' campus technology use.
31

2015 GHP Evaluation Report
did not agree as strongly as Music majors (4.6) that communication with their RA was easy. Mathematics (4.3), Social Studies (4.2), and World Languages (4.3) majors all agreed less than Music majors (4.6) that they almost always used their personal device(s) to connect to the internet.
In relation to GHP courses and whether the courses offered a unique learning experience, Communicative Arts (4.9) and Social Studies (4.8) majors agreed more than Music majors (4.6) that their GHP major offered a unique learning experience. Alternatively, Dance (4.3) and Agricultural Science (3.7) majors agreed less than Music majors when asked the same question. As for the GHP elective, Theatre Performance (3.9) majors were the only group to differ from Music majors (4.5) about the uniqueness of their GHP elective.8
When asked about the GHP instructors, Dance (3.7), Agricultural Science (3.9), and World Languages (4.4) majors all agreed less than Music majors (4.7) that their GHP instructors challenged them. Compared to Music majors (4.7), Dance (4.0), Agricultural Science (4.3), Visual Arts (4.4), and World Languages (4.4) majors agreed less that their GHP instructors helped them to be successful; meanwhile, Communicative Arts majors (4.9) agreed more when asked the same question. When asked about their perceptions of GHP instructors' tolerance of diverse opinions, only Communicative Arts majors (4.9) agreed more than Music majors (4.6) that GHP instructors were tolerant.
Student responses differed by major when they were asked about their future plans. Music majors (4.1) agreed more than Dance (3.5), Social Studies (4.0), and World Languages (3.4) majors when asked if their experiences at GHP helped them to learn more about careers related to their interests. When they were asked if their GHP major will influence their college major or minor, Music majors (4.0) agreed more than World Languages majors (3.3). Finally, Communicative Arts majors (4.8) tended to agree more than Music majors (4.3) that they will be able to use what they learned at GHP in other high school classes.
Differences by Race Compared to White students (4.6), African American students (4.1) agreed less when asked if they almost always used their personal device(s) to connect to the internet. Asian (4.5) and Multi-Racial (4.3) students agreed less than White students (4.7) that their GHP major offered a unique learning experience. Asian students (4.4) agreed less than White students (4.7) that their GHP instructors helped them to be successful. In terms of diversity, compared to White students (4.4), only African American students (3.7) agreed less that the GHP faculty is appropriately diverse. Additionally, African American students (3.6) agreed less than White students (4.6) that the GHP student body is appropriately diverse.
8 For questions related to the GHP Elective, student responses were analyzed based on their GHP major because GHP students took more than one GHP Elective during the program. Furthermore, administrative data does not contain students' GHP Elective selections.
32

2015 GHP Evaluation Report

When asked about their GHP major's influence on a college major or minor, Asian (3.6), Multi-Racial (3.4), and African American (3.4) students all agreed less than White students (4.1) that their GHP major was influential. Finally, in terms of high school usefulness, only Asian students (4.2) agreed less than White students (4.5) that what they learned at GHP will be useful in other high school classes.
Open-Ended Responses

The 2015 GHP student evaluation survey consisted of mostly scaled response items. However, a three of the questions allowed students a space to write freely. The three questions were (1) why they would or would not recommend GHP to a friend; (2) the factors that influenced their decision to come to GHP; and (3) if there was anything else to add.

Though more than 97% of survey respondents would recommend GHP to a friend, their reasons for doing so take many forms. Many students simply said that GHP was "fun", "magical", and one of the best experiences of their life. Many students noted the personal development that GHP offers through a unique learning environment, engaging instructors and access to a broader array of talented and passionate students than they interact with at home. Several students also brought up the point that GHP is free of charge and that this facet is a benefit. In total, fifteen students (<3%) indicated that they would not recommend GHP to a friend (see Table 19). A few of the students who would not recommend GHP to a friend gave reasons for withholding the recommendation. Two Science majors and one World Languages major cite the lack of intellectual stimulation and low level of performance from other students in their major.

Table 19: Student Survey Respondents to Recommend GHP

Number of

Percentage of

Respondents

Respondents

No

15

<3%

Yes

531

97%

Total

546

100%

When asked about the factors that influenced their decision to attend GHP, most students stated the program's reputation and attending GHP would boost their college applications or resume. A second major factor is the recommendations they received from GHP alumni, with many alumni being the student's own sibling, a close friend, or a classmate. A third major influence is the encouragement of their teachers and parents. A fourth major factor is the lack of cost to attend GHP. Answers to this survey item include the most references to the cost of GHP. A fifth major factor is the student's own passion and their desire to learn and practice what interests them. One Mathematics student explains it as an "opportunity of a lifetime to be with likeminded students who actually care about learning." A Communicative Arts major writes that she came because "for some weird reason

33

2015 GHP Evaluation Report
[she wanted] to learn over the summer." Many students described their motivation as an opportunity or desire to challenge themselves and meet new people.
Though it did not specifically ask about how GHP can be improved, the final openended response item garnered several suggestions for how to improve GHP. The top two suggestions were to remove the mandatory concerts and the 7:00 PM hall checks. Several students, including some Music majors who performed at the event, made reference to the mandatory concerts. Most students who complained about the mandatory concerts did not like being forced to attend the concerts, and some suggested that the turnout would still be good for the events if students were given the choice to attend or not attend. As one Science major wrote, "I feel that even if [the concerts] weren't mandatory they would still have a large turn out. The fact that they were mandatory was a source of discontent throughout the program and made people not want to go. As a musician myself in my free time, I can truly say I would rather have an audience that chooses to be there and listen than one that is completely disinterested and counting the minutes to leave." Most of the concert performers who mentioned the mandatory concerts did not feel like the audience enjoyed, or even paid attention to, their performance. Some students felt underappreciated because their own majors did not have mandatory events like the concerts. A Music major commented that "the fact that concerts were mandatory, yet coffee houses, dance recitals, and theatre performances were not, lead many students to being frustrated with the system and that they were not as equally represented to the many students attending the program."
The 7:00 PM hall checks were also a point of contention in the last survey item. Most students felt like the 7:00 PM hall checks interrupted their daily activities and social life. Specifically, a few Performing Arts majors commented on how disruptive the 7:00 PM hall checks were to their practice times because they had to travel across campus from their performance areas to their dorm rooms.
Student Survey Summary
Several notable differences exist between groups in the student evaluation survey data. These differences are noteworthy because they constitute a 0.7 or more difference in the average scores submitted by different student groups. Most of these big differences occur between specific majors, but big differences are also found between major area and race. First, African American students differed drastically from their White counterparts on issues of diversity, both within the faculty and within the student body. African American students average 3.7 out of five--compared to 4.4 out of five for White students--when asked if the GHP faculty is appropriately diverse. The gap widens when African American students are asked to think about student diversity (3.6 versus 4.6). African American students may be accustomed to seeing a more diverse faculty and student body at their normal school, or African American students may hold an entirely different view of diversity. The difference between scores may occur for any number of
34

2015 GHP Evaluation Report
reasons, but without any detail from African American students, all explanations are circumspect at best.
Performing Arts majors tended to agree more than Academic majors that the GHP coursework is more rigorous than what they normally experience (4.2 versus 3.5). Again, a multitude a reasons may influence their opinion. For example, being a Performing Arts major could be more physically taxing than being an Academic major, and the GHP environment allows students to pursue their interests more intensely. The difference could be from the way Performing Arts majors structure their day with practice and rehearsals. The difference could also be due to the location of the classes and structure of GHP. On campus, most Performing Arts majors' classes were located across campus, further away from the dorm rooms and dining hall than Academic majors' classes. The back and forth during the day, as well as the physical nature of their majors, could also play a role in the perceptions of Performing Arts majors.
Most of the noteworthy differences--when the difference between two categories of students is 0.7 points or more--occur by specific major. When compared to the perception of Music majors, who make up the largest proportion of GHP students, the opinions held by Dance, Agricultural Science, and World Languages majors offer stark differences to the norm. Dance majors held significantly lower opinions than Music, Communicative Arts, Engineering, Mathematics, Social Studies, and Theatre Performance majors on how much instructors challenged them artistically. The differences here are interesting because, unlike Agricultural Science and World Languages, Dance is categorized as a Performing Art, just like Music. The differences between Dance and Music majors may be routed in the type of performance that is required from each or the amount of structured time that each major requires. The differences could signal deeper disparities in intrinsic motivations between students, or the differences could even be a result of the levels of talent that students perceived within their major.
Agricultural Science majors differed a lot from Music majors when it came to how much they learned in their major (3.6 versus 4.4), being academically or artistically challenged at GHP (3.4 versus 4.7), feeling challenged by their instructors (3.9 versus 4.7), and the rigor of the coursework (3.3 versus 4.2). These differences could be explained by the available technology and facilities on the college campus, compared a student's normal high school. Many high-achieving and highly sought after STEM-focused high schools have science labs to rival college and university campus labs. The differences could also spur from the level of instruction or the level of talent that the students perceive within their own majors.
World Languages majors also differed from the norm in interesting areas. World Languages majors did not think the coursework was as rigorous (3.3 versus 4.2), that their GHP major helped them learn more about careers related to their interests (3.4 versus 4.1), or that their GHP major influenced their future college major or minor (3.3 versus 4.0). The second two differences are interesting because they
35

2015 GHP Evaluation Report
both relate to future plans with a student's GHP major. If giving students insight into potential college majors and postsecondary career options is a priority for GHP, then administrators can work to provide that guidance for all students.
One theme dominated the students' suggestions for how to improve GHP. In essence, GHP students want more freedom to choose their evening activities. Several students expressed disdain for the mandatory concerts and the 7 PM hall checks. Student patrons and performers at the mandatory concerts both suggested that the other group, and whole concert experience, would benefit by filling the concert hall with individuals that want to attend or perform, and allowing uninterested individuals to forego the event. This line of reasoning is also echoed in the faculty/staff and RA evaluation surveys.
Students also do not like the 7 PM hall checks. Hall checks occur at 7 PM and 10:30 PM during the week and on weekends, with one additional hall check on Saturdays and Sundays. In the student survey, no respondents suggested removing the 10:30 PM hall check or the weekend hall checks. However, the 7 PM hall check was portrayed as an unnatural time constraint that interrupted the academic, extracurricular, and social activities that happen after dinner.
FACULTY/STAFF EVALUATION SURVEY
The 2015 GHP Faculty/Staff Evaluation Survey consisted of 25 questions. The questions were organized by topic area, which included, in order of appearance: general information (2), training (4), administrative support (5), materials and resource (3), satisfaction and diversity (7), and demographics (4). Except for the first question, which instructed respondents to enter their 9-digit VSUID number, all of the survey items were optional. Sixteen of the survey items were 5-point Likert scale questions, six of the items were drop-down selections, two items were solely open response, and one item was for the VSUID numbers. All of the survey items, except the drop-down responses and the required response, included a prompt and space for comments.
One of the priorities of the current GHP administrators is to collect and encourage feedback from the program's faculty and staff. Due to the importance of feedback, the faculty/staff evaluation survey items included many open response portions. It was also important to collect some demographic information from survey respondents, data that the current administration did not already have. Unfortunately, without a demographic profile of all faculty and staff, there is no way to estimate if the survey respondents are representative of the full population. Survey respondents were asked to supply demographic information--such as their race/ethnicity, gender, years taught at GHP, etc.--which cannot be matched to any current GHP administrative data. For example, survey data shows that 30% of faculty/staff respondents were female and 70% were male. Without administrative
36

2015 GHP Evaluation Report

data showing the proportion of females to males in the full faculty/staff population, this evaluation cannot estimate whether females were proportionally represented in the survey data, overrepresented in the survey data, or underrepresented in the survey data.

The faculty/staff evaluation survey was sent to faculty and staff via email. Faculty and staff received one initial invitation to complete the survey and then three reminder emails to complete the survey. Like the students, faculty and staff received notice of the survey during the final week of GHP. The faculty/staff evaluation survey remained accessible for a total of three weeks.

Faculty/Staff Survey Responses

The 2015 GHP faculty and staff evaluation survey was sent to all 57 faculty and staff members. A total of 34 faculty and staff responded to the survey, which is a response rate of about 56%. Because little administrative data exists on faculty and staff, it is difficult to infer that the survey respondents reflect the total GHP faculty and staff population.

Table 20 shows the faculty and staff respondents by gender. Of the faculty and staff who responded, 9 (30%) were female and 21 (70%) were male. Table 21 is shows that the majority of respondents were White (63%) or African American (13%). Nearly 10% of respondents choose not to answer, 6% identified as Asian, and 9% identified as either Latino, Multi-Racial, or Native American. The majority of survey respondents were either Staff (14%) or Music instructors (14%), followed by about 12% each in Communicative Arts, Social Studies, and World Languages, and nearly 9% in Visual Arts (see Table 22). At least one faculty member from each GHP major, with the exception of Theatre, responded to the survey.

Table 20: Faculty/Staff Respondents by Gender

Frequency Percent

Female

9

30%

Male

21

70%

Total

30

100%

37

2015 GHP Evaluation Report

Table 21: Faculty/Staff Respondents by Race/Ethnicity

Frequency Percent

African American/Black

4

13%

Asian/Pacific Islander

2

6%

Latino/Hispanic

1

3%

Multi-Racial Native American/Alaskan Native Prefer not to answer

1

3%

1

3%

3

9%

White, non-Hispanic

20

63%

Total

32

100%

Table 22: Faculty/Staff Respondents by Major

Frequency

Music

5

Communicative Arts

4

Dance

2

Engineering

1

Mathematics

2

Agricultural Science

2

Science

1

Social Studies

4

Theatre Tech

1

Visual Arts

3

World Languages

4

Staff

5

Total

34

Percent 14% 12% 6% 3% 6% 6% 3% 12% 3% 9% 12% 14% 100%

Respondents were asked to select their industry, where they work outside of GHP. Table 23 reports the selections. Seventy-five percent of respondents selected Education, Training, and Library as their occupational industry. Fourteen percent selected Arts, Design, Entertainment, Sports, and Media. The remaining 11% of respondents came from Architecture and Engineering, Healthcare Support, and Life, Physical, and Social Science.

38

2015 GHP Evaluation Report

Table 23: Faculty/Staff Respondents by Industry

Frequency Percent1

Architecture and Engineering

1

4

Arts, Design, Entertainment, Sports, and Media

4

14

Education, Training, and Library

21

75

Healthcare Support

1

4

Life, Physical, and Social Science

1

4

Total

28 100%

1Percentages have been rounded and may not add up evenly to 100%.

GHP has been operational for over 50 years, and many of its faculty and staff return each year to teach a new cohort of students. The GHP administrative data does not contain information on the number of years each faculty or staff member has taught at GHP. Survey respondents were asked to select the number of years that they have taught at GHP years taught. The timeframes, along with their percentages, are displayed in Table 24. The answer choices included four categories: 1 year, 2-5 years, 6-10 years, or more than years. Of survey respondents who answered, 35% have taught at GHP for only one year, 29% have taught at GHP for 2-5 years, 13% have taught at GHP for 6-10 years, and 23% have taught at GHP for more than years.

Table 24: Faculty/Staff Respondents by Years Taught at GHP

Frequency

Percent

1 year

11

35%

2-5 years

9

29%

6-10 years

4

13%

More than 10 years

7

23%

Total

31

100%

Faculty/Staff Survey Analysis
Analysis of the faculty/staff evaluation survey is limited for two main reasons. First, as aforementioned, little administrative data exist on the GHP faculty and staff population. The lack of data from the full population of GHP faculty/staff makes it difficult to draw inferences on the full population from the survey sample. Second, the total number of responses to the faculty/staff evaluation survey is low. Without responses from a variety of different faculty and staff, it becomes difficult to make comparisons between groups.
The following analyses of the faculty/staff evaluation survey focuses on differences between average scores by faculty/staff groups, namely by major area and by years

39

2015 GHP Evaluation Report
taught, and presents the averaged survey item responses. An analysis of openended responses is also included. The two groups used for means testing, major area and years taught, were selected because the number of respondents in each category would be greater than three and each category's score can be computed as a mean average. Another reason why the number of individuals in each category is important is to protect the confidentiality of the survey respondents. Major area includes two groups of faculty/staff, Academic instructors or Performing Arts instructors. Years taught, includes four groups of faculty/staff divided by years of teaching at GHP: 1 year, 2-5 years, 6-10 years, and more than 10 years. A full table of the survey items is available in Appendix B.
Means Testing
ANOVA tests were conducted to look for statistically significant differences between the two groups of major area, Performing Arts instructors and Academic instructors. Test revealed that there are no significant differences in the answers given by Performing Arts instructors to the answers given by Academic instructors. Since no statistically significant differences exist, the conclusion is that the overall average score for each of the five-point Likert scale survey items can be understood to represent the overall average score for all Instructors.
ANOVA tests were also conducted to look for differences between the four groups of years taught: 1 year, 2-5 years, 6-10 years, and more than 10 years. Like the round of tests conducted by major area, no statistically significant differences exist between scores submitted by instructors who have each taught at GHP for a varying number of years. Therefore, the conclusion is that the overall average score for each of the five-point Likert scale survey items can be understood to represent the overall average score given by all faculty and staff, regardless of how many years of teaching the faculty or staff member has with GHP.
Table 25 presents each five-point Likert scale survey item and the item's overall score. The average scores can be understood to reflect the average opinion of all survey respondents, not the full GHP faculty and staff population. The relatively lower scores for questions related to training and materials and resources is worth noting. Faculty and staff training occurred a few days before students arrived on campus. Most of the survey respondents have undergone the same or similar training events each year that they taught at GHP. The lower scores for training could be a result of participant maturation, or the level of familiarity that corresponds with the length of time that a faculty or staff member has taught at GHP. The lower scores for materials and resources, however, could be directly attributed to the new ordering process and the issues with delivery. When developing the evaluation instruments, GHP administrators expressed concern over how orders and deliveries were handled this year. Specifically, GHP administrators experienced logistical difficulties in confirming deliveries and they wanted to know if a similar frustration was felt on the part of the faculty and staff.
40

2015 GHP Evaluation Report

Table 25: Faculty/Staff Average Scores for Scaled Responses

Survey Question
Training
1. The training that I received in June for GHP helped me throughout the duration of the program.
2. The material presented during the June training helped me to be a more effective GHP instructor.
3. The faculty/staff handbook was a useful tool throughout the duration of GHP.

Avg. Score1 3.4 3.4 3.4

Support

Avg. Score

4. The GHP Program Manager supported me throughout the duration of GHP.

4.2

5. My GHP department chair supported me throughout the duration of GHP.

4.8

6. The GHP Dean of Instruction supported me throughout the duration of GHP

4.8

7. The instructional support that I received from the Dean of Instruction was helpful to me during GHP.

4.5

8. The observational feedback that I received from the Dean of Instruction was helpful to me during GHP.

4.2

Materials and Resources

Avg. Score

9. I am satisfied with the ordering process for instructional materials.

3.8

10. I am satisfied with the delivery process for instructional materials.

3.5

11. My students had all of the necessary resources they needed to produce a high-quality Final Project.

4.0

Satisfaction and Diversity

Avg. Score

12. I am satisfied with my experience at GHP.

4.3

13. I would like to teach at GHP again.

4.7

14. The students that I taught were highly-talented

4.3

15. The GHP faculty is appropriately diverse.

4.2

16. The GHP student body is appropriately diverse.

4.2

1The average score is based on a five-point scale where 1=Strongly Disagree and

5=Strongly Agree

Open-Ended Responses
An analysis of the open-ended responses submitted by faculty and staff provides greater detail of what they thought, both positive and negative, about training, support, materials and resources, satisfaction, and diversity. In regards to training,

41

2015 GHP Evaluation Report
many veteran respondents commented that the training material was not useful or new to them. Most respondents enjoyed the time to interact with their colleagues, plan curricula, and prepare their classrooms. One Academic instructor commented, "As a returner, I don't know that any of the information was particularly new to me from the pre-planning meetings. However, I thoroughly enjoy the camaraderie of my colleagues, the ability to set up classrooms, and collaboration with my department during this time."
Faculty and staff scores for the support they received from the Dean of Instruction are the highest scores in the survey. In their comments, faculty and staff speak specifically about what the Dean of Instruction did well. One Academic instructor writes:
I bristle a bit at the formal observation structure that has been introduced into GHP this year because it feels so much like school, and often at school, we are observed by administrators with less teaching experience and by people who don't want to be in the classroom - it's hard to swallow advice from administrators like this... What I like a lot about [the Dean of Instruction's] feedback is that she really paid careful attention to what I was doing and gave me points to improve on that actually reflected careful thinking and strong pedagogical insights. Her feedback led me to have conversations with my fellow teachers about how I could always improve, change my instruction for next year to make it excellent. To me, that's really significant - we talk a lot about the kids and encouraging them to take chances, experience constructive feedback, and grow. I really believe that [her] approach helps foster this possibility for the faculty.
A deeper look at the comments provided in the materials and resources section of the survey, shows that what frustrated faculty and staff most about the ordering and delivery process is that it was neither streamlined nor immediate. For example, one instructor from Performing Arts remarked that "the ordering process was more complicated than last year." One staff member suggested that the "ordering process could be streamlined and done electronically." The GHP program lasts for four weeks. It is imperative for instructors to have all of their necessary materials as soon as the program begins. As one Academic instructor noted, "the length of the [GHP] program does not afford much wait time. When supplies are needed, two week gaps are not ideal."
When asked to comment about satisfaction, several instructors mentioned the effects of the students' mandatory meetings and concerts. One Academic instructor writes, "The mandatory meeting times for students and mandatory programs severely cut into students' abilities to engage with the classwork provided them - articles went unread, which caused some vital discussions to fall flat. As an adaptable, resourceful professional, I can combat this in the class, but I felt my work was undermined when students did not have adequate time to access/engage/reflect upon the work provided them..." Another Academic
42

2015 GHP Evaluation Report
instructor alludes to the mandatory events as "severe restrictions on [students'] non-instructional time." One Performing Arts instructor suggests that the concerts and performances will be equally well attended even if they are not mandatory, and that "students need time to rest and reflect" between instruction and performance.
Most faculty and staff responded to the question about faculty diversity in terms of race. One Academic instructor writes that there are "too many white men like me." Another instructor cannot recall seeing any other African American female instructors at GHP this year. An Academic instructor states that "the faculty is mostly white. That's not diversity." However, the lack of faculty diversity is not a uniformly shared sentiment among respondents. One staff member writes that there is a "wonderful collection of folks from all backgrounds, cultures, [and] professional settings." In response to the question about student diversity, faculty and staff wrote about diversity in more than racial terms, such as gender diversity and geographic diversity. For example, a Performing Arts instructor thinks that "there could be more males in the dance program." An Academic instructor suggest that "more effort should be made to encourage non-Metro schools to nominate and prepare candidates for the program." Though one Academic instructor does feel that "we need to make a real effort to include Latinos in the program," her suggestion is "to work on [it] at a local level at [her] school and [her] county."
Faculty/Staff Survey Summary
Findings from the faculty/staff evaluation survey only reflect the views of the survey respondents because not enough information is known about the full faculty/staff population to say that the survey respondents represent all faculty and staff. This evaluation, and future evaluations, would benefit from a faculty and staff demographic profile. Going forward, GHP may want to consider ways to build the collection of demographic data into its operational model.
Faculty and staff survey respondents generally agreed on what worked for them in the current GHP model and what sort of changes they would like to see going forward. First, the majority of respondents found the administrative support and observational feedback provided by the Dean of Instruction to be pedagogically useful. Responses remained consistent across gender, major, and years taught. The administrative support, especially the observational feedback, should continue.
Faculty and staff, however, were less approving of the on-site training they received prior to students arriving on campus for GHP. Many respondents found the most useful portion of the on-site training to be the time that they had to work within their major and network with other colleagues. Many veteran GHP instructors did not see the utility of sitting through the training again. Instead, GHP may want to consider restructuring the training sessions or staggering when in their years taught with GHP new and old instructors must attend a training session.
43

2015 GHP Evaluation Report
The faculty and staff respondents also offer a suggestion that is reverberated in the student and RA evaluation surveys: removing the mandatory concerts. Respondents who raised this point argued that GHP students need to be given more freedom of choice. They also argued that mandatory concerts cut into the limited study and practice time that students had available. One respondent argued that the mandatory concerts could be detrimental student relations by preferencing and showcasing one major or set of majors above all others.
The faculty and staff responses to questions of diversity indicate areas for more direction. Respondents mention racial and gender diversity as areas where it is lacking for faculty, and racial, gender, and geographic diversity as areas where it is lacking for students. The program could use its historic data, where possible, to highlight areas of homogeneity and then focus recruiting efforts to attract a more diverse group of students and instructors.
RESIDENTIAL ADVISOR EVALUATION SURVEY
The 2015 RA Evaluation Survey was the shortest of the three surveys, and consisted of 26 questions. Like the other two surveys, the only required answer was the first question which asked for respondents to enter their 9-digit VSUID number. The questions were organized by topic area, which included in order of appearance: general information (2), training and preparation (6), administrative support (6), the student experience (4), satisfaction (4), and demographics (4). Except for the first question, which instructed respondents to enter their 9-digit VSUID number, all of the survey items were optional. Eighteen of the survey items were 5-point Likert scale questions, six of the items were drop-down selections, one was solely open response, and one item was for the VSUID numbers. All of the survey items, except the drop-down responses and the required response, included a prompt and space for open response.
During the design of the RA evaluation survey, GHP administrators expressed concern with the RAs perceptions of preparedness, support, and communication. Therefore, the RA evaluation survey focused on GHP-provided trainings, time allotments for preparing activities, and feedback loops between the RAs and Dean of Residential Life. Similar to the faculty/staff evaluation survey, little demographic information for RAs exists in GHP's administrative data, though the RAs' gender can be inferred by their housing assignment. However, no inferences about the larger RA population are possible from the data collected during the evaluation.
The RA evaluation survey was sent to RAs via email. RAs received one initial invitation to complete the survey, and then two reminder emails afterwards. The RA evaluation survey remained accessible for a total of three weeks.
44

2015 GHP Evaluation Report

RA Survey Responses
The 2015 GHP RA evaluation survey was sent to all 35 Residential Advisors, which includes two Dorm Directors and thirty-three Residential Advisors. A total of 13 RAs responded to the survey, which is a response rate of about 37% (see Table 26). This is the lowest response rate of the three evaluation surveys. Like the faculty/staff evaluation survey results, little administrative data exist on RAs, therefore it is difficult to infer that the survey respondents reflect the total GHP RA population.

Table 26: RA Survey Response Rate

Frequency Percent

No Response

22

63%

Responded

13

37%

Total

35 100%

Table 27 shows that, of the thirteen RAs who responded, 8 (62%) were female and 5 (38%) were male. All of the respondents identified as White, non-Hispanics. The majority of survey respondents served as RAs (92%) and one served as a Dorm Director (see Table 28).

Table 27: RA Survey Respondents by Gender

Frequency

Percent

Female

8

62%

Male

5

38%

Total

13

100%

Table 28: RA Respondents by Position Frequency Percent

Dorm Director

1

8%

RA

12

92%

Total

13

100%

GHP administrators described the RAs as being either college students, teaching assistants, or teachers. GHP requires RAs to live on campus, in the dorms, amongst the GHP students. Most GHP RAs tend to be or work in the field of education where summer breaks are a common occurrence. With the job requirements in mind, the survey asked respondents to select an occupation that best describes them. Table 29 displays the results. Half of respondents chose "college student with no Residential Life experience," and the other half of respondents was equally split between "college student with Residential Life experience," "graduate student," and "teaching professional."

45

2015 GHP Evaluation Report

Finally, GHP administrators also acknowledged that, like faculty and staff, many RAs return to GHP each year to work with the program. The evaluation survey asked respondents if they are GHP alumni, to which two-thirds of respondents answered yes (see Table 30).

Table 29: RA Respondents by Occupation

Frequency

College student with Residence Life experience

2

College student with no Residence Life

6

Graduate student

2

Teaching professional

2

Total

12

1Percentages have been rounded and may not add up to 100.

Percent1 17% 50% 17% 17% 100%

Table 30: RA Respondents by Alumni Status

Frequency Percent

Not alumnus/a

4

33%

Alumnus/a

8

67%

Total

12 100%

RA Survey Analysis
Analysis of the RA evaluation survey is limited for the same reasons as the faculty/staff evaluation survey analysis is limited. First, very little administrative data exist for the full population GHP RAs, making it difficult to draw inferences to the total RA population for the survey's small sample. Second, the response rate is low, as is the variety of respondents, which makes it difficult to create comparison groups from the data.
The analysis for the 2015 GHP RA evaluation survey consists of means testing by groups, a presentation of overall scores, and a brief analysis of the open-response survey items. Group comparisons are by gender and by alumni status. These two groups were selected because they offer the most variety in respondents while still being large enough to protect respondents' confidentiality. Gender includes males and females, which is a category created by the respondent's residence hall selection. Alumni status also includes two categories from a question in which respondents marked yes or no. A full table of the survey items is available in Appendix C.

46

2015 GHP Evaluation Report
Means Testing One-way ANOVA tests were conducted to look for statistically significant differences on the five-point Likert scale survey items by either gender or alumni status. The tests revealed that there are significant differences in the answers supplied by male RAs to the answers supplied by female RAs. Additionally, no differences in answers were found between alumni RAs and non-alumni RAs. Since no statistically significant differences exist, the conclusion is that the overall average score for each of the scaled responses can be understood to represent the overall average score given all RAs, regardless of gender or alumni status. Table 31 presents each five-point Likert scale survey item and the item's overall score. The average scores can only be understood to reflect the overall opinions of the survey respondents, not the entire GHP RA population. Average scores were relatively high for questions related to communication with the Dorm Directors, and relatively low for questions related to time for preparation and students' social lives at GHP. The higher scores for Dorm Directors could be due to the structure of GHP's Residential Life division. The thirty-three residential advisors each report to one of two Dorm Directors. The Dorm Directors are responsible for moving information up and down the Residential Life administrative pipeline, so it is important for them to keep open lines of communication with the RAs and the Dean of Residential Life. GHP administrators forewarned that RAs may not have felt that the pre-planning time was sufficient because RAs had a lot of tasks to complete before the program started. The assumption appears to hold true and those feelings were also echoed in the open-response submissions. Context for the low scores in student socialization are also echoed in the open-response submissions.
47

2015 GHP Evaluation Report

Table 31: RA Average Scores for Scaled Responses

Survey Question

Training and Preparation

Avg. Score1

1. The training that I received in June helped prepare me to advise GHP students.
2. The June training at GHP was similar to what I received as an RA at my college/university.
3. The faculty/staff handbook was a useful tool throughout the duration of GHP.
4. The Residence Life handbook/training manual was a useful tool throughout the duration of GHP.
5. My role and responsibilities were clearly communicated to me before arriving on campus.
6. I had enough time to plan and prepare seminars before the GHP students arrived on campus.
Administrative Support
7. I felt supported by my Dorm Director throughout the duration of GHP.
8. It was easy to talk to my Dorm Director about any issue.
9. I received feedback from my Dorm Director in a timely and appropriate manner.
10. I felt supported by the Dean of Residence Life throughout the duration of GHP.
11. It was easy to talk to Dean of Residence Life about any issue.
12. I received feedback from the Dean of Residence Life in a timely and appropriate manner.
The Student Experience

4.0 3.3 3.7 3.8 3.7 2.7 Avg. Score1 4.9 4.9 4.7 4.5 4.3 4.5 Avg. Score1

13. This summer, GHP students got a good glimpse of what living

3.8

on a college campus is like.

14. GHP students need more help navigating the social life at

2.7

GHP.

15. The scheduled Residence Life activities were appropriate for

3.9

GHP students.

16. The GHP student body is appropriately diverse.

4.5

Satisfaction

Avg. Score1

17. I am satisfied with my experience at GHP.

4.8

18. I would be a Resident Advisor/Dorm Director at GHP again.

4.6

1The average score is based on a five-point scale where 1=Strongly Disagree and

5=Strongly Agree

48

2015 GHP Evaluation Report
Open-Ended Responses
For each content area, RAs were asked to provide details about their scaled scores. Most of the RAs thought that the training prepared them to do their job, but also that no amount of training could prepare them for everything. Two female RAs write that "there were some things that we couldn't have prepared for" and "I was trained for as much as I could be..."
Many of the respondents mentioned being unclear about their job responsibilities, specifically as they relate to seminar planning and when they would have time to plan seminars. The pre-planning time that RAs did have during orientation was mostly spent decorating the halls and not planning seminars and activities. One female RA writes, "During orientation, there is no time for seminar planning. All of our extra time (not in meetings) is spent putting our hall decorations together and up on the walls." In order to complete all of their ongoing responsibilities, many RAs used their off-time during the program to plan seminars. A dorm director notes, "...I had no time to plan seminars, either during orientation week, or during the program except during off-time. This was also the case for most of the RAs." One male RA also commented saying, "Seminar time at night was not clearly expressed as a time where we couldn't plan seminars." Based on this feedback, Residential Life may need to consider building ongoing planning times for RAs into the program.
RAs were not only concerned about being able to plan seminars; they were also concerned about students being able to attend seminars. The issue regarding mandatory events and 7 PM hall checks was also apparent in the RA survey evaluation, like it was in the student and faculty/staff survey evaluations. These two events were specifically mentioned as reasons why students do not get a college experience at GHP; as well as reasons why GHP students do not need more help navigating the social life, but more freedom to be social. One female RA writes, "The only help GHP students need socially is given to them through the hosting of seminars by RAs. In a college setting, students have absolute freedom, and at GHP they do not. This year in particular, they had less freedom of experience because of the extra hall checks and mandatory events." A dorm director writes,
GHP students don't need more help navigating the social life at GHP than what we provide, but they do need more time to cement those bonds. The students were able to make friends in their majors, minors, and on the halls, but did not have the time, nor the freedom to make meaningful bonds with others in the greater GHP community...[T]he lack of prep time for RAs and the continuing demand on the students' time due to mandatory events meant a reduction in the offerings of seminars and activities... GHP students are incredibly driven and need to have time to enjoy themselves and make friends outside of an academic setting. They are in class for 6 1/2 hours every day...
49

2015 GHP Evaluation Report
RA Survey Summary Although the response rate and number of responses for the RA evaluation survey were low, one common concern amongst respondents is the need for more time to plan Residential Life seminars. One RA did not know what to expect in terms of time commitment. Another RA misunderstood when he would have time to plan his seminars. Two RAs expressed a need for ongoing time, throughout the duration of GHP, to plan seminars and activities. In the future, the Residential Life division of GHP might consider devising structured, ongoing planning time for the RAs. Finally, respondents offered similar sentiments to those found in the student and faculty/staff evaluation surveys. RAs did not like the students' mandatory events. From a Residential Life perspective, one dorm director suggested that cutting back on the students' mandatory events might allow students more time to engage with each other during the seminars and activities.
50

2015 GHP Evaluation Report
Conclusion
The 2015 GHP evaluation covers the second year that the program is being administered by GOSA. Below are some of the major themes from each evaluation survey, as well as the most common theme throughout the evaluation.
Student Evaluation The analysis of the pre-post question pairs revealed that student expectations may not be fulfilled after attending GHP. The difference in scores for pre-post questions should be examined by student subgroups to see if there are systematic processes that GHP could implement to increase students' perceived rigor and level of learning. Although many students mentioned coming to GHP to learn and better their future, a lot of them also mentioned socializing with other students. Some students did not feel as though they had enough time to socialize with their peers or pursue their own interests due to the demands mandatory activities. In the future, GHP may consider building in more loosely-structured social time for students to meet each other and make connections outside of their GHP major. Finally, students were very vocal about their dissatisfaction with the 7 PM hall checks and the mandatory events. Students expressed discontent with these two specific event types mainly because these events take away their ability to choose what they do in their free time.
Faculty/Staff The faculty and staff evaluation found some interesting highlights within GHP. First, the majority of surveyors responded very favorably to the Dean of Instruction and the observational feedback she provides. Faculty/staff who did comment mentioned incorporating observational feedback into their teaching styles. Second, the training seemed to be most useful to novice or newer GHP instructors. However, all instructors appreciated the time to plan lessons and meet with their colleagues before students arrive on campus. In the future, GHP may consider offering a staggered training schedule to allow returning instructors to bypass GHP-sponsored trainings and go right into the lesson plans and collaborating with colleagues. Instructors also spoke the lack of diversity within GHP. Racial diversity, gender diversity, and geographic diversity were all areas for improvement highlighted by faculty and staff.
Residential Advisors The biggest concerns for GHP RAs is the amount of time they have to plan and prepare before students arrive on campus and the amount of time they have during the program to plan new lessons. The Residential Life staff do meet regularly during GHP and GHP students are in class every morning for six days per week. To improve, GHP should consider finding time for RAs to plan new seminars, at the beginning on of the program and during the program.
51

2015 GHP Evaluation Report
The most widespread themes throughout this year of GHP are the calls from students, faculty/staff, and RAs to abolish the mandatory events. Many respondents from each participant group talked about how the mandatory concerts limit students' freedom, unjustly preference some majors above others, and changed the mood of GHP.
52

2015 GHP Evaluation Report
Appendix A: 2015 GHP Student Evaluation Survey Items For the following survey, students were asked to either (1) enter information, (2) select from a list of options, or (3) select their level of agreement to the statement. Levels of agreement were presented in a five-point Likert scale where 1=Strongly Disagree, 2=Disagree, 3=No Opinion, 4=Agree, 5=Strongly Agree, and NA=Not Applicable.
Table 32: 2015 GHP Student Evaluation Survey Items General Information 1. Please enter your 9-digit VSUID number, without any spaces. 2. *What is your GHP Major? (list of Majors) 3. What is your GHP Elective? (list of Electives)
Program Rigor 4. I expected to learn something new at GHP that would be helpful to me later in life. 5. I learned something new at GHP that will be helpful to me later in life. 6. I expected to perform at a level higher than I usually do at my high school. 7. At GHP, I performed at a level higher than I usually do at my high school. 8. At GHP, I learned something new that I was not expecting to learn. 9. *GHP engaged me academically/ artistically. 10. GHP engaged me socially. 11. GHP coursework is more rigorous than what I normally experience. 12. Overall, I am satisfied with GHP.
Residential Life 13. I expected to have a wide variety of appropriate activities to choose from outside of my
major and elective requirements. 14. The campus facilities (e.g., classrooms, study areas, meeting/practice rooms, etc.)
helped create a positive learning environment for me. 15. The Residence Life seminars and activities improved my GHP experience. 16. It was easy to communicate with my RA. 17. I felt safe on the VSU campus. 18. I almost always used my own device(s) to connect to the internet.
Courses 19. I expected to learn a lot from my GHP Major. 20. I learned a lot from my GHP Major. 21. I expected to learn a lot from my GHP Elective. 22. I learned a lot from my GHP Elective. 23. My GHP Major offered a unique learning experience.
53

2015 GHP Evaluation Report
24. My GHP Elective offered a unique learning experience. Instructors
25. My GHP instructors appropriately challenged me. 26. My GHP instructors helped me be successful. 27. The GHP faculty was appropriately diverse. 28. The GHP instructors were tolerant of diverse opinions. 29. I know who the counselors are and how to reach them. 30. The GHP counseling services were useful to me.
Future Plans 31. Before coming to GHP, I knew I had all of the skills necessary to succeed in college. 32. My experiences at GHP helped me learn more about careers related to my interests. 33. I plan to continue learning more about my GHP Elective. 34. My GHP Major influenced my decision on a college major or minor. 35. My GHP Elective influenced my decision on a college major or minor. 36. I will be able to use what I learned at GHP in other high school classes.
Other 37. I made the right decision to come to GHP. 38. GHP has an appropriately diverse student body. 39. Attending GHP posed a financial hardship on my family. 40. GHP students were tolerant of diverse opinions. 41. Would you recommend GHP to a friend? Why or why not? 42. What factors influenced your decision to come to GHP?
Demographics 43. Please select your high school from the list. 44. Will you be a junior or senior in the upcoming school year? 45. Is there anything you would like to add? *An asterisk indicates a question filtered by major area. 1Note. This number includes duplicate responses. The actual number of responses per question varies. 2Note. Percentages for "no opinion" have been universally excluded.
54

2015 GHP Evaluation Report

Table 33: Scaled Response Average Scores for All Students Survey Question
Program Rigor 1. I expected to learn something new at GHP that would be helpful to me later in life.
2. I learned something new at GHP that will be helpful to me later in life.
3. I expected to perform at a level higher than I usually do at my high school.
4. At GHP, I performed at a level higher than I usually do at my high school.
5. At GHP, I learned something new that I was not expecting to learn.
6. GHP engaged me academically/ artistically. 7. GHP engaged me socially. 8. GHP coursework is more rigorous than what I normally experience.
9. Overall, I am satisfied with GHP.
Residential Life 10. I expected to have a wide variety of appropriate activities to choose from outside of my major and elective requirements. 11. The campus facilities (e.g., classrooms, study areas, meeting/practice rooms, etc.) helped create a positive learning environment for me. 12. The Residence Life seminars and activities improved my GHP experience. 13. It was easy to communicate with my RA. 14. I felt safe on the VSU campus.
15. I almost always used my own device(s) to connect to the internet.
Courses
16. I expected to learn a lot from my GHP Major. 17. I learned a lot from my GHP Major.
18. I expected to learn a lot from my GHP Elective. 19. I learned a lot from my GHP Elective. 20. My GHP Major offered a unique learning experience.
21. My GHP Elective offered a unique learning experience.
Instructors
22. My GHP instructors appropriately challenged me. 23. My GHP instructors helped me be successful. 24. The GHP faculty was appropriately diverse. 25. The GHP instructors were tolerant of diverse opinions.

Avg. Score1 4.7
4.5
4.6
4.3
4.6 4.5 4.6 3.7 4.5 Avg. Score 4.5
4.4
4.2
4.6 4.4 4.5 Avg. Score 4.7 4.4 4.4 4.3 4.6 4.6 Avg. Score 4.6 4.6 4.3 4.6

55

2015 GHP Evaluation Report

26. I know who the counselors are and how to reach them.

3.9

27. The GHP counseling services were useful to me.

3.3

Future Plans

Avg. Score

28. Before coming to GHP, I knew I had all of the skills necessary to

3.5

succeed in college.

29. My experiences at GHP helped me learn more about careers related

3.9

to my interests.

30. I plan to continue learning more about my GHP Elective.

4.2

31. My GHP Major influenced my decision on a college major or

3.8

minor.

32. My GHP Elective influenced my decision on a college major or

3.3

minor.

33. I will be able to use what I learned at GHP in other high school

4.4

classes.

Other

Avg. Score

34. I made the right decision to come to GHP.

4.8

35. GHP has an appropriately diverse student body.

4.5

36. Attending GHP posed a financial hardship on my family.

1.6

37. GHP students were tolerant of diverse opinions.

3.8

1The average score is based on a five-point scale where 1=Strongly Disagree and

5=Strongly Agree

56

2015 GHP Evaluation Report
Appendix B: 2015 GHP Faculty/Staff Evaluation Survey Items For the following survey, faculty and staff were asked to either (1) enter information, (2) select from a list of options, or (3) select their level of agreement to the statement. Levels of agreement were presented in a five-point Likert scale where 1=Strongly Disagree, 2=Disagree, 3=No Opinion, 4=Agree, 5=Strongly Agree, and NA=Not Applicable.
Table 34: 2015 GHP Faculty/Staff Survey Items General Information
1. Please enter your VSUID number, without any spaces. 2. *Please select your position: Faculty or Staff June Training 3. *Which Major/Elective area did you teach in? (See lists below) 4. The training that I received in June for GHP helped me throughout the
duration of the program. Please explain 5. The material presented during the June training helped me to be a more effective GHP instructor. Please explain 6. The faculty/staff handbook was a useful tool throughout the duration of GHP. Please explain Administrative Support 7. The GHP Program Manager supported me throughout the duration of GHP. Please explain 8. My GHP department chair supported me throughout the duration of GHP. Please explain 9. The GHP Dean of Instruction supported me throughout the duration of GHP Please explain 10. The instructional support that I received from the Dean of Instruction was helpful to me during GHP. 11. The observational feedback that I received from the Dean of Instruction was helpful to me during GHP. Materials and Resources 12. I am satisfied with the ordering process for instructional materials. Please explain 13. I am satisfied with the delivery process for instructional materials.
57

2015 GHP Evaluation Report
Please explain 14. My students had all of the necessary resources they needed to produce a
high-quality Final Project. Please explain Satisfaction and Diversity 15. I am satisfied with my experience at GHP. Please explain 16. I would like to teach at GHP again. Please explain 17. Would you recommend GHP to a colleague as a good place to work? Why or Why not? 18. The students that I taught were highly-talented. Please explain 19. The GHP faculty is appropriately diverse. Please explain 20. The GHP student body is appropriately diverse. Please explain 21. Is there anything else you would like to add? Demographics 22. Including this year, how many years have you taught at GHP? (4 options: 1, 2-5, 6-10, more than 10) 23. What is your current occupation outside of GHP? (See list) 24. What is your gender? 25. Please select your race/ethnicity. *An asterisk indicates a filtered question. Faculty will choose from a list of Majors and Staff will choose from a list of Electives.
58

2015 GHP Evaluation Report
Appendix C: 2015 GHP RA Evaluation Survey Items For the following survey, residential advisors (RAs) were asked to either (1) enter information, (2) select from a list of options, or (3) select their level of agreement to the statement. Levels of agreement were presented in a five-point Likert scale where 1=Strongly Disagree, 2=Disagree, 3=No Opinion, 4=Agree, 5=Strongly Agree, and NA=Not Applicable.
General Information 1. Please enter your 9-digit VSUID number, without any spaces. 2. *Please select your position. (Resident Advisor, Dorm Director) 3. The training that I received in June helped prepare me to advise GHP students. 4. The June training at GHP was similar to what I received as an RA at my college/university. 5. The faculty/staff handbook was a useful tool throughout the duration of GHP. 6. The Residence Life handbook/training manual was a useful tool throughout the duration of GHP. 7. My role and responsibilities were clearly communicated to me before arriving on campus. 8. I had enough time to plan and prepare seminars before the GHP students arrived on campus.
Administrative Support 9. *I felt supported by my Dorm Director throughout the duration of GHP. 10. *It was easy to talk to my Dorm Director about any issue. 11. *I received feedback from my Dorm Director in a timely and appropriate manner. 12. I felt supported by the Dean of Residence Life throughout the duration of GHP. 13. It was easy to talk to Dean of Residence Life about any issue. 14. I received feedback from the Dean of Residence Life in a timely and appropriate manner.
The Student Experience 15. This summer, GHP students got a good glimpse of what living on a college campus is like. 16. GHP students need more help navigating the social life at GHP. 17. The scheduled Residence Life activities were appropriate for GHP students. 18. The GHP student body is appropriately diverse.
Satisfaction 19. I am satisfied with my experience at GHP.
59

2015 GHP Evaluation Report
20. I would be a Resident Advisor/Dorm Director at GHP again. 21. Would you recommend GHP to your colleagues as a good place to work?
Why or why not? 22. Is there anything else you would like to add? Demographics 23. Are you a GHP alumnus/a? 24. In which residence hall did you work? 25. Which of the following best describes you? (College student with
Residence Life experience, College student with no Residence Life experience, Teaching professional, Graduate student) 26. Please select your race/ethnicity. *An asterisk indicates questions filtered by staffing selection--either Dorm Director or Resident Advisor.
60