Data utilization study [Nov. 1999]

Data Utilization Study

November 1999
Prepared by Margaret H. H. Brackett
Gary T. Henry Jeanie Weathersby

Buford Arnold Dorothy Cottom Melvin Kruger

Prepared for The Council for School Performance

Pat Willis, Chairman

Leena Johnson Dean Swanson Linda Schrenko, Ex-officio

Jeanie Weathersby, Director Gary T. Henry, Research Director

Data Utilization Study
Executive Summary
The Council for School Performance has conducted the first phase of a Data Utilization Study to investigate how Georgia's schools and school systems use the School and School System Performance Reports produced by the Council. The information collected from this study will be utilized to improve the content and distribution of the Reports so as to ensure that they provide the most useful information.
While the Council for School Performance directs and distributes the annual Performance Reports to a wide audience of policymakers, practitioners, and others, the Reports primarily serve school and school system officials. For the first phase of this study, the Council surveyed a sample of principals and school system contacts about their use of the reports. Data collected from these surveys reveal that the vast majority of both groups consider themselves familiar or very familiar with the Performance Reports. Ninety-eight percent of system contacts and ninetyfive percent of principals characterize themselves as familiar or very familiar with the Council Reports. At the same time, system contacts and principals act as an important link in distributing the Reports to a broader audience. The majority of both groups share the Reports with other system and school officials, teachers, and parents. Of the 96% of the principals who distribute the Reports to others, 91% share them with teachers.
Having familiarized themselves with the Council's Reports, the school system contacts and principals participating in the study consider the data in the Performance Reports to be meaningful and useful. The respondents answered questions about three areas of use for the Reports: (a) as a tool for assessment, (b) as a basis for discussion, and (c) as an impetus for policy changes. Of these three areas, system contacts and principals report that they use the Reports most extensively for assessment and comparison purposes. They utilize Council data most extensively to compare their system/ school with others in the state, to compare their system/ school with similar systems/ schools in the state, and to evaluate progress over time. System contacts and principals have found the Council's unique "cluster" system--grouping schools and systems into groups according to similar demographic characteristics--to be especially helpful; they cited comparison with systems/ schools with similar demographic characteristics as the most extensive application of the Performance Reports.

Data Utilization Study
Introduction
In 1993, the Council for School Performance was created with the mission to provide impartial and accurate information regarding the performance of public schools and lottery-funded education programs. To that end, the Council produces annual Performance Reports on Georgia's more than 1,800 schools and school systems. In addition to distributing these reports throughout the state, the Council also conducts workshops for school and school system personnel to increase their understanding of the interpretation and analysis of school performance data.
This year marks the fourth annual release of the Performance Reports, and the Council has launched the first phase of a Data Utilization Study to investigate how the School and School System Performance Reports are used. The information collected from this study will be utilized to assess the content and distribution of the Reports so as to ensure that they provide the most useful information.
For the first phase of this study, the Council surveyed a random sample of principals and school system contacts about their use of the reports.1 During the spring of 1999, a survey questionnaire was mailed to 133 principals and 133 system contacts throughout the state (see Appendix A for principal survey and Appendix B for system contact survey). Seventy-four (55.6%) of the principals and 96 (72.2%) of the system contacts completed and returned the surveys. The respondents answered questions about three areas of use for the Reports: (a) as a tool for assessment, (b) as a basis for discussion, and (c) as an impetus for policy changes.
The system contact persons were selected as the survey respondents--rather than the superintendents--because the system contacts act as the main consumers and distributors of the information in the Performance Reports at the school system level. In this position, system contacts have the ability to assess the information in the Performance Reports from a systemwide perspective, while also determining what school-level information impacts the system overall.
At the school-level, principals from the different school levels were surveyed. Of the principals responding to the survey, 70% are from elementary schools, 14% from middle schools, 15% from high schools, and 1% from primary schools.2 Principals' proximity to students and teachers, in conjunction with their accountability to parents and school system officials, places them in the unique position of handling multiple responsibilities in areas as diverse as academic performance and community participation. From this perspective, they can utilize the information in the School Performance Reports differently than system contacts--including assessing the utility of the information at the school-level.
1 More detailed information about the respondents and the response rate is included in the "Methodology" section. 2 School-level is determined by designation according to the School Performance Reports.

Data Utilization Study - 2
Findings
Familiarity with and Distribution of Performance Reports
The Data Utilization Study reveals that the Council for School Performance successfully reaches the system contacts with the Performance Reports. The majority of the system contacts (73%) declare themselves very familiar with the Reports, and another 25% view themselves as familiar with the Council's Reports. Only 2% of those surveyed report that although they have seen the Reports, they have never made use of their information.
Principals also indicate familiarity with the Council's Reports, but to a lesser extent than the system contacts. Just over half of the principals (53%) view themselves as very familiar with the Reports, and another 42% consider themselves familiar with the Reports.3 Only 5% of the principals say that they have seen the Reports but have not used them.
System contacts act as a vital link in the distribution of the Performance Reports to other school officials and the public. Although the Council itself mails the Reports to every school in Georgia, the system contacts--with additional insight into the immediate concerns and objectives of local school officials--also distribute the Reports and frequently highlight indicators of interest to the system or to an individual school. Eighty-nine percent of the system contacts distribute the Reports to others. Like the system contacts, the principals also act as an important part of the distribution network for the Council's Reports. With their firsthand knowledge of a school, they understand which indicators most interest parents and teachers. As Figure 1 shows, 96% of the principals distribute the Reports to others.
Figure 1. Percentage of System Contacts and Principals Who Distribute the Performance Reports to Others

Principals

96%

System Contacts

89%

0%

20%

40%

60%

80%

100%

3 The percentages of principals responding to these questions are the percentages of the entire sample of principals (from all school levels).
Council for School Performance

Data Utilization Study - 3
As indicated by Figures 2 and 3, system contacts and principals distribute the Performance Reports to others, with both groups targeting teachers as their main audience for distributing the Performance Reports. A high percentage of system contacts also include the information in newsletters, while a majority of principals also share the information with parents.4
Figure 2. How System Contacts Share the Performance Reports with Others

Distribute to Teachers

40%

Include in Newsletter

22%

Distribute to Parents

16%

0%

20%

40%

60%

80%

100%

Figure 3. How Principals Share the Performance Reports with Others

Distribute to Teachers

91%

Distribute to Parents

51%

Include in Newsletter 0%

30%

20%

40%

60%

80%

100%

4 These percentages reflect the proportions of system contacts and principals who indicated that they distribute the Reports to others, rather than the proportion of all system contacts or principals in the sample. For example, of the 89% of system contacts who distribute the Reports to others, 40% send them to teachers. These percentages do not sum to 100 since respondents could select more than one option for distribution.
Council for School Performance

Data Utilization Study - 4
As a part of its efforts to support ongoing school improvement efforts, the Council conducts workshops on using the Performance Reports for school officials and community leaders throughout the state. Through these workshops, individuals increase their ability to interpret and analyze their school performance data. Almost one-third (32%) of the system contacts report having attended one of the Council's workshops. Principals appear less likely to have had the opportunity or to have taken the opportunity to attend the Council's workshops, with only 11% of the principals reporting attendance.
Perception of Usefulness of Performance Reports
Overall, school system contacts feel that the Performance Reports are useful for their school systems and for their communities. Fifty-two percent of the system contacts and 45% of the principals indicate that the Reports have impacted education in their systems and schools. The majority of the system contacts and the principals--79% and 71%, respectively--report that the Council's data provide meaningful information about "the quality and effectiveness" of their school systems and their schools.
The respondents view the Performance Reports as useful information for local decision-makers as well as for the community in general. Almost two-thirds of the system contacts (64%) and almost half of the principals (49%) think that the Council's Reports offer information that can be used for resource allocation decisions. At the same time, 33% of the system contacts and 34% of the principals report that Council information has increased community understanding of educational progress in their school systems and schools. Increased understanding does not necessarily coincide with more support for schools, according to the survey results. Only 15% of the system contacts report that the Performance Reports have strengthened support for the school system within the community, while 27% of the principals attribute increased community support for their schools to the Reports.
How the Performance Reports Are Used
The survey questions the school system contacts and the principals about three areas of use for the Council's Performance Reports: (a) as a tool for assessment, (b) as a basis for discussion, and (c) as an impetus for policy changes. The extent to which the system contacts and principals utilize the Reports is measured along a 5-point scale, with "1" indicating "never" using the Reports for that purpose, and "5" indicating "very extensive" use of the Reports for that purpose. Overall, how extensively the system contacts and principals have used the Reports varies. The Council's data are used most extensively for assessment purposes, while also providing information as a basis for discussion and policy changes.
According to the answers of the system contacts and the principals, the Council Reports are utilized extensively for assessment and comparison purposes. As Figure 4 depicts, system contacts and principals use Performance Reports most extensively to compare their system or schools with others in the state, to assess strengths and weaknesses, and to evaluate progress over time. The data collected reveal that system contacts and principals appreciate the Council's unique "cluster" system, which allows comparison among schools and systems with populations sharing similar demographic characteristics. Using the Reports for this purpose rates the highest average "extent of use" at both the system and school levels. System contacts and principals take advantage of the cluster information, with 61% of system contacts and 55%
Council for School Performance

Data Utilization Study - 5

of principals reporting that they use Council data extensively (4 or 5 on the scale) to compare system or school results with similar systems--demographically--in the state.
Figure 4. How Extensively System Contacts and Principals Use Performance Reports For Assessment (on average)

To Compare with Similar Schools/ Systems Demographically

3.53 3.55

To Compare with Other Schools/ Systems in the
State

3.44 3.38

To Evaluate Progress Over

3.43

Time

3.43

To Assess Strengths and Weaknesses Regarding
Student Behavior
To Assess Strengths and Weaknesses Regarding
Academics

3.33 2.79
3.25 3.08

To Assess Curriculum

3.22 2.85

To Evaluate Instructional Programs

2.84 2.75

To Create a Database

2.52 2.35

1 Never

2

3

Council for School Performance

System Contacts Principals

4

5

Very Extensively

Data Utilization Study - 6
In addition to using the Reports for assessment purposes, system contacts and principals also use them to promote discussions regarding the quality and effectiveness of schools. Roughly one-third of system contacts report having discussed the Performance Reports with different groups. Based on the average "extent of use," both system contacts and principals use the Reports most extensively in their discussions with teachers. The system contacts also utilize the Reports extensively in their communications with the school board. Figures 5 and 6 reveal that the system contacts and the principals use the Reports to different extents with the various audiences with whom they interact. Both system contacts and principals also use the Council Reports extensively to identify performance areas that require additional discussion and review; both groups indicate an average "extent of use" of 3.55 (on the 5-point scale) for this purpose.
Figure 5. How Extensively System Contacts Use Performance Reports in Discussions (on average)

With Teachers

3.16

With School Board

3.05

With Local Media

2.81

With Personnel Dept.

2.80

1

2

Never

3

4

5

Very Extensively

Figure 6. How Extensively Principals Use Performance Reports in Discussions (on average

With Teachers

3.45

With Parents

2.73

With Local Media

2.01

1

2

3

Never

Council for School Performance

4

5

Very Extensively

Data Utilization Study - 7
In addition to applying the Reports for assessment and discussion purposes, Council data have been employed extensively to promote change at the school- and system-level. As Figure 7 reveals, system contacts and principals indicate that they utilize the Performance Reports most extensively as a tool for change when they use the information to support school improvement plans and to initiate plans for improving student performance. They use Council data least extensively in teacher evaluations--resulting in an average "extent of use" of less than 2.0 for both groups of school officials. Only 4% of system contacts and 4% of principals indicated extensive (4 or 5 on the scale) use of the Reports for teacher evaluations. Almost half of system contacts (49%) and principals (45%) use the Council's Reports extensively to support school improvement plans.
Figure 7. How Extensively System Contacts and Principals Use Performance Reports to Promote Change (on average)

To Support School Improvement Plans
To Initiate Plans for Improving Student Performance on Specific Indicators
To Set Goals for Improving Student Performance on Specific Indicators
To Support Changes in Changes in Curriculum
To Support Changes in Educational Policies
1 Never

3.30 3.04

3.17 2.85

3.09 2.85

2.90 2.55

2.75 2.38

2

3

System Contacts Principals

4

5

Very Extensively

Council for School Performance

Data Utilization Study - 8
Methodology This study was conducted by the Council for School Performance, through the Applied Research Center at Georgia State University. From the population of all school principals, school system contact persons, and PTA/ PTO presidents at public schools in Georgia, 133 of each group were selected at random to receive the survey. Surveys were mailed in late March 1999. Each survey package included an introductory letter explaining the intention of the study, a short questionnaire, a sample (generic) School or System Performance report, and a prestamped, pre-addressed envelope for returning the survey. Reminder postcards were mailed to respondents one and one-half weeks after the surveys were initially sent. Upon receiving the reminder postcard, some individuals called to request additional copies of the survey because they had misplaced or had never received the initial mailing. Of the 133 principals, 74 (55.6%) completed surveys and returned them to the Council. Of the 133 school system contact persons, 96 (72.2%) completed surveys. Of the 133 PTA/ PTO presidents, 16 completed surveys (12.0%) and returned them to the Council. The low response rate from the PTA/ PTO presidents prohibits inclusion of their responses in the report at this time. The lack of response from this group suggests that many of the targeted respondents never received the original survey, or that newly elected officers chose not to respond to the survey because of their recent transition to their office. The surveys were not addressed to these individuals by name but were sent to "The PTA/ PTO President" at each school. Few additional requests for surveys were received from this group, further supporting the theory that these individuals did not receive the initial survey.
Council for School Performance

Appendix A

Council for School Performance Data Utilization Survey SCHOOL LEVEL

SCHOOL

Thank you for taking time to complete this survey. This evaluation is designed to assess the extent to which the Council for School Performance Reports (see enclosed sample) are used in school-level decisions and program improvements. We encourage you to provide as complete answers as you can to each question.
When completing this survey, please be assured that the information provided by you will be reported in the aggregate; none of the results will be reported by individual school or school system.

1. How familiar would you consider yourself with the Council for School Performance reports (see enclosed sample)?

m Very familiar; have used them often m Familiar; have seen and used them at least once m Not familiar; have seen them but never used them m Do not recognize at all

2. Have you or a member of your staff ever attended a workshop on using the Council's Reports?

Yes

No

Don't Know

3. Each year, the Council for School Performance mails copies of the School Performance Reports to principals, superintendents, system contact people, PTA/ PTO Presidents, School Board chairpersons, and Chambers of Commerce. When you receive your copy, do you make it available to others?

Yes

No

Don't Know

If Yes, How? Please check all that apply Distributed to Teachers Distributed to Parents Include information in newsletter Other ________________

Council for School Performance

4. On a scale of 1 to 5, where 1 is Never and 5 is Very Extensively, please assist us in determining the extent to which your school has used Council for School Performance Reports.

Never

Very Extensively

Do the reports...

1

2

3

4

5

provide meaningful information about the quality and effectiveness of my school

have increased community understanding of educational progress in my school

have increased community support for education in my school

provide information that local decision makers can use for resource allocation

have had little impact on education in my school

5. Please assist us in determining the ways in which your school has used Council for School Performance reports.

We have used the council for school performance reports:

Strongly Agree Agree

Strongly Don't Neutral Disagree Disagree Know

to establish a database

as a basis for discussion with teachers

as a basis for discussion with parents

as a basis for discussion with local media

to compare results for your school with those of other schools throughout the state

to compare results for your school with those similar schools (demographically) in the state

to evaluate progress from one year to the next

to identify areas that warrant further discussion and review

to assess the effectiveness of curriculum

to assess investment in teacher development

Council for School Performance

5. Please assist us in determining the ways in which your school has used Council for School Performance reports. (continued from previous page)

We have used the council for school performance reports:

Strongly

Strongly

Agree Agree Neutral Disagree Disagree

Don't Know

to assess the effectiveness of instructional programs/ strategies

to provide information for the evaluation of teachers

to assess strengths and weaknesses of student performance (regarding academics)

to assess strengths and weaknesses of student performance (regarding dropout, absenteeism, etc.)

to initiate plans of action to improve student performance on specific indicators

to set targets/ goals for student performance on specific indicators

to support changes in curriculum

to support school improvement plans

to support changes in educational policies

Council for School Performance

Please address each of the following questions. 6. Do you have any suggestions of how indicator data for the Council for School Performance (CSP)
reports could be more useful for your school? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 7. What are some additional indicators that would increase the use of CSP reports for school accountability and/ or for school improvement? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 8. Additional comments: ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
Thank you for completing this survey!
You may access individual School and System Performance Reports via our website, http://arcweb.gsu.edu/csp
Council for School Performance

Appendix B
Council for School Performance Data Utilization Survey SYSTEM LEVEL

SCHOOL

Thank you for taking time to complete this survey. This evaluation is designed to assess the extent to which the Council for School Performance Reports (see enclosed sample) are used in system-level decisions and program improvements. We encourage you to provide honest and complete answers to each question.
When completing this survey, please be assured that the information provided by you will be reported in the aggregate; none of the results will be reported by individual school or school system.

1. How familiar would you consider yourself with the Council for School Performance reports (see enclosed sample)?

m Very familiar; have used them often m Familiar; have seen and used them at least once m Not familiar; have seen them but never used them m Do not recognize at all

2. Have you or a member of your staff ever attended a workshop on using the Council's Reports?

Yes

No

Don't Know

3. Each year, the Council for School Performance mails copies of the School Performance Reports to principals, superintendents, system contact people, PTA/ PTO Presidents, School Board chairpersons, and Chambers of Commerce. When you receive your copy, do you make it available to others?

Yes

No

Don't Know

If Yes, How? Please check all that apply Distributed to Teachers Distributed to Parents Include information in newsletter Other ________________

Council for School Performance

4. On a scale of 1 to 5, where 1 is Never and 5 is Very Extensively, please assist us in determining the extent to which you have used Council for School Performance Reports.

Never

Very Extensively

Do the reports...

1

2

3

4

5

provide meaningful information about the quality and effectiveness of my school

have increased community understanding of educational progress in my school

have increased community support for education in my school

provide information that local decision makers can use for resource allocation

have had little impact on education in my school

5. Please assist us in determining the ways in which your school has used Council for School Performance reports.

We have used the council for school performance reports:

Strongly Agree Agree

Strongly Don't Neutral Disagree Disagree Know

to establish a database

as a basis for discussion with teachers

as a basis for discussion with the school board

as a basis for discussion with local media

to compare results for your school with those of other schools throughout the state

to compare results for your school with those similar schools (demographically) in the state

to evaluate progress from one year to the next

to identify areas that warrant further discussion and review

to assess the effectiveness of curriculum

to assess investment in teacher development

Council for School Performance

5. Please assist us in determining the ways in which your school has used Council for School Performance reports. (continued from previous page)

We have used the council for school performance reports:

Strongly

Strongly

Agree Agree Neutral Disagree Disagree

Don't Know

to assess the effectiveness of instructional programs/ strategies

to provide information for the evaluation of teachers

to assess strengths and weaknesses of student performance (regarding academics)

to assess strengths and weaknesses of student performance (regarding dropout, absenteeism, etc.)

to initiate plans of action to improve student performance on specific indicators

to set targets/ goals for student performance on specific indicators

to support changes in curriculum

to support school improvement plans

to support changes in educational policies

as a basis for discussion with the personnel department of our school system

as a basis for discussion with parents

Council for School Performance

Please address each of the following questions. 6. Do you have any suggestions of how indicator data for the Council for School Performance (CSP)
reports could be more useful for your school? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 7. What are some additional indicators that would increase the use of CSP reports for school accountability and/ or for school improvement? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 8. Additional comments: ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
Thank you for completing this survey!
You may access individual School and System Performance Reports via our website, http://arcweb.gsu.edu/csp
Council for School Performance