Getting started with program evaluation : a guide for arts organizations

Getting Started with Program Evaluation
A Guide for Arts Organizations

Contents
Why evaluate ..........................................................2 In addition to demonstrating accountability to your funders and stakeholders evaluation can strengthen your organization by helping you to improve programs solve problems and make a compelling case for more resources.
What is "data" and where do I get it ..............3 Useful data is all around you. It is embedded in the response of your audience and many tools you may already use to manage your work. Choose a thoughtful variety of information--quantitative and qualitative--to tell your story and measure the outcomes of your programs.
Can what we do in the arts really be measured ..................................................................5 Yes. Both the artistic and business dimensions of your work can be meaningfully evaluated although your methods may need to be as creative as your programming
Evaluation frameworks..............................................................6 Logic models and rubrics are powerful planning tools that also can help you organize your evaluation efforts.
Advice for the beginner.....................................9 Following some practical "dos" and "don ts" can help any arts organization get started with evaluation.
Cultivating a learning culture ....................10 Make the most of your evaluation efforts by engaging your constituents staff and board.
Recommended resources..................................11 Consult these resources for additional information and ideas.
ACKNOWLEDGEMENTS............................................12

Untitled Tom Graffagnino Georgia State Art Collection
This guide is a joint production of the Georgia Council for the Arts and the National Assembly of State Arts Agencies.
2007. Reproduction and dissemination of this document is encouraged provided the Georgia Council for the Arts (GCA) and the National Assembly of State Arts Agencies (NASAA) receive attribution and copyright citation.
Zebra Beach IV Patrick McCay Georgia State Art Collection

Why evaluate
What methods will you use to evaluate this program
How will you monitor the effects of your program on participants
What indicators will you use to measure your progress in attaining your goals
Phrases like these will be familiar to many arts organizations especially those that seek funding from public agencies or foundations. If your organization has not yet encountered an evaluation requirement of some kind get ready Evaluation has become a fact of life for nonprofit organizations in the 21st century. Like all other charitable groups arts and cultural organizations are increasingly required to document what they accomplish and assess the impact of their programs.
When combined with good planning practices however evaluation is not just a reporting requirement imposed upon arts organizations by funders. It is also a tool for organizational learninga tool that can help manage your programs spend your funds wisely and help your constituents and potential investors understand your work.
This guide is designed to help arts organizations reap these evaluation benefits. Its intent is to provide introductory information and ideas that arts organizations may supplement with their own experience as well as with the growing body of literature about program evaluation in the arts. Most of all this guide is designed to invite your creative thinking and to encourage you to use evaluation to tell your story in authentic and compelling ways.

Enhance the results of your programs marketing efforts or management. Evaluation can shed light on persistent challenges and reveal keys to success. It can also help you pinpoint what strategies provide the greatest return on your time and effort.
Help grant-makers and donors understand your work. Evaluation results illustrate the impact of your programs. This information helps funders describe the community benefits that stem from their sponsorship. It also can help them justify future funding. Evidence of ongoing evaluation further shows that good planning and management practices are in place in your organization which raises confidence among all of your investors.
Demonstrate accountable use of taxpayers dollars. State federal and local governments bear additional responsibilities for ensuring that public dollars are being put to good use. When you receive funding from a state arts agency the National Endowment for the Arts or any municipal or county agency you become a steward of that public trust too. Evaluation helps to demonstrate that the taxpayers dollars you receive are in good hands.
Engage your stakeholders. Program participants audience members staff volunteers artists and teachersall have insight to share. Involving stakeholders in your evaluation conveys that you value their input and reinforces connections among the people taking part in your program.
Help communities recognize the value of the arts. Suzanne Callahan in Singing Our Praises Case Studies in the Art of Evaluation writes that research "builds trust understanding and visibility for the arts field convincing those in the outside world of the value of what we do." (Callahan p. 15) By evaluating their programs conducting research and sharing knowledge with others arts organizations deepen public appreciation for the role that the arts play in society.
Make the case for more resources. Evidence of what you are currently accomplishingand what you could accomplish with additional resourcesstrengthens your requests for financial human and material assistance. This is particularly important in the public sector where funding is scarce and the arts must compete with a host of other pressing needs. Your evaluation efforts can help you and others advocate for the arts and prove that supporting culture is in the best interests of your state s citizens children families businesses and communities.
2

What is "data" and where do I get it
If you can observe it it is data.
Data is factual information especially information that is systematically organized and used to help someone make a decision. Most people associate the word "data" with numbers dollars demographics percents or averages. Statistical or standardized information usually called quantitative data is part of most evaluations but is by no means the only kind of information that is useful to arts organizations. While quantitative data can identify important trends or patterns qualitative data is often needed to reveal the deeper meaning behind the numbers.
For instance an audience survey can quantify the percent of people who might attend another performance at your venue. Until you probe for additional qualitative information however little about that data is actionable for you as an arts manager. Asking "Why or why not " to elicit qualitative responses may reveal important insight such as
The parking is terrible I could not hear the dialogue I only go to events when I can get a
babysitter I received this ticket as a gift and
cannot afford to buy another or I faithfully attend every choral event in
our town.
Armed with such information you can now decide which conditions you wish to address in the future. This is why a good evaluation typically blends different kinds of information quantitative information to identify major trends and qualitative information to provide depth of understanding.
Untitled Patricia Hetzler Georgia State Art Collection

ARTS DATA A SAMPLER OF IDEAS
Participation data describes your audience and their experiences.
Number of individuals participating Participant demographics Home zip code or address Motivation for attending
Opinions about the event or exhibit Personal arts interests and preferences Participant ratings of your customer service
Perceived barriers to participation Etc...
Marketing data helps you attain your audience development goals and understand the effectiveness of your outreach.
Where did participants hear about this event Were tickets purchased via mail on-line or in person
What types of tickets were purchased Were outreach strategies to target markets successful
Etc...
Financial data can help you calculate management efficiency and return on investment.
Admissions income Memberships
Grants and contributions Number of donors
Number of donors increasing their contributions Project costs
Savings achieved Etc...
Impact data helps you understand the effects of your program on participants and your surrounding community.
Perceived benefits to participation Changes in behaviors or attitudes
Learning outcomes School climate
Achievement attendance or discipline of students Audience spending on parking meals or housing
Quality of life Etc...
3

Outcomes Measurable changes in a person or group s status skills attitudes knowledge or circumstances.

Many different sources of data are available to arts organizations. A few examples include

Outcome goals
The changes you hope to achieve in your organization in your community or among your program s participants. Outcome goals can be

Admissions/enrollment records Many arts organizations track ticket sales or class enrollments through a database or spreadsheet program. Consider

short-term or long-term. It is helpful to articulate them in your project plan or your organization s strategic plan.

how you might mine this data in order to measure the extent of participation in your events. Are numbers increasing over time holding steady or declining

Indicators

Why Do you systematically collect and record all the

Selected dataquantitative or qualitativethat

information that you and your funders need Assess

can be gathered to measure progress toward your outcome goals. You can choose any indicators but make sure that they are both relevant to the outcome and feasible for you to collect.

whether you should secure additional information to tell you more about your audience and assist you with marketing your programs.

Financial records Try to tally all financial

transactions (including salaries) connected to your

project. How do your expenses compare with your revenue What

areas are most expensive but worth every penny In what areas

might cost-savings be achieved Do you have specific financial

goals for your organization or program How close are you to

achieving those goals When developing your budgets at the

beginning of your fiscal year be sure they contain enough detail for

you to realistically reflect on these questions and make sound

financial management decisions.

Surveys Surveys especially audience surveys are common evaluation and marketing tools in the arts field. Often they are used to understand the composition of an organization s audience but they also can be adopted to gather information on opinions motivations preferences perceived barriers or other issues. The success of surveys usually hinges on two factors the quality of the questions (crafting questions that are clearly understood and can be accurately answered) and the respondent pool (securing a response that is sufficiently large and diverse to be truly representative of your population).

Untitled Maurice Blaine Caldwell Georgia State Art Collection

Interviews Interviews may be conducted in person or on the telephone and are a good choice when the information you need is sensitive or confidential in nature or when the respondent will appreciate personal attention. Interviews also offer you the flexibility to adapt the inquiry to individuals by omitting irrelevant questions or taking more time to probe key issues. To encourage candid responses choose an interviewer who will be perceived as trustworthy by the interviewees.

Focus groups Focus groups are designed to elicit qualitative information from a small number of people through dialogue and group interaction. Five to ten individuals typically participate in a focus group which are often used to probe the "why s and wherefores" of participants behaviors or opinions. Focus groups are best led by a trained moderator an impartial individual who is

4

knowledgeable about the subject and skilled in group facilitation techniques. If you want to conduct a focus group you will need to develop a protocol in advance to ensure that the conversations address the questions that matter most to you. Arrange for a transcript of the proceedings to enable analysis of the information.
Pre- and post-tests When designed around specific learning objectives tests can measure the "before and after" skills knowledge or perceptions of students in workshops classes or other educational programs. A comparison of these two scores is one way of measuring the learning and skill outcomes of arts instruction.
Journals or portfolios Journals or portfolios provide a written or visual record of learners experiences and achievements. Artists and arts teachers often cite the dual value of these assessment techniques. They not only help instructors measure skill development but also are tools that encourage students to self-reflect on their own work.
Direct observations Observation is a useful way to gather real-time behavioral information using artists teachers staff or volunteers. Observers can be trained to look for specific actions activities or behaviors during a class performance or other event. Consistency is the key to observation success. Be sure that all observers are well-practiced and attentive. Provide a uniform checklist or log that helps them record what they see.
Can what we do in the arts really be measured

"Can we measure joy Yes we can. And by knowing where and how to look for it how to document what we see and hear and how to communicate that we can create opportunities for more joy more often."

These are the words of Deborah Bedwell director of Baltimore Clayworks a nonprofit ceramic arts center housed in Baltimore Maryland. Challenged by funders to evaluate the impact of their community arts program the Clayworks staff decided to investigate not only the delivery of their services but also the effects of hands-on clay classes on local youth. To do this Clayworks staff developed a list of indicators teachers would be able to observe and record among students when an authentic engaging experience was taking place. A few of those items included

Shows work to peers Concentrates on techniques Holds work close to body Uses clay vocabulary

Clayworks also tracked students enrollment their completion of ceramic works their completion of journals and whether they recruited friends and family members to participate in subsequent events and

classes. Combined this information painted a powerful portrait of the program s success. Using a logic model for their program also helped Clayworks clearly articulate their intended

Basket Kathy Gottlieb Georgia State Art Collection

outcomes and focus their evaluation efforts where they would provide the most useful feedback for

planning and further program improvement.

5

Similar stories can be shared by many arts organizations who have found creative ways to assess the arts participation experience of their constituents. None of these organizations is seeking to quantify the transcendent nature of the artistic experience. However an evaluation that focuses exclusively on the mechanics of an arts program may leave a significant gap in our understanding of its full value and impact.

One useful way to add an artistic dimension to an evaluation is to brainstorm what might be observable about a participant s experience. For instance
How did individuals participate in your event Did they stay for the entire event Did they attend multiple times Did they watch attentively Did they become physically involved Did they take advantage of all participation opportunities such as interactive exhibits audio tours or artist discussions

An evaluation that focuses exclusively on the mechanics of an arts program may leave a significant gap in our understanding of its full value and impact.

What reactions did your work evoke among participants What did they like the best Like the least What feelings ideas or questions did their experience evoke

What do you hope participants will learn about the art form or the artistic process and how could that be measured Can students use the required tools materials or instruments Do they demonstrate mastery of certain exercises or techniques Can they express key concepts Do students exhibit creativity by engaging in experimentation or pursuit of their own original ideas

What else might participants learnabout themselves their fellow participants or the larger communitythrough your program Inviting personal reflections from participants can be a rich source of qualitative information.

What did the artists performers instructors or other project staff experience What did they learn themselves What reactions did they observe in others

Remember that the most meaningful measures will be those that are authentic to your unique goals constituency and art form. There are no "right" or "wrong" indicators to monitor. Measure what is relevant to you and aligns best with your program s goals and objectives.

Evaluation frameworks

Rubrics and logic models are two planning tools that can provide helpful structure for the evaluation of any arts program.
Widely used in the education sector as a tool for measuring student success in classroom settings rubrics are a tool that many arts organizations have adapted to their unique arts programming and management environments. Simply defined a rubric is a grid that articulates what project success and failure look like in concrete terms. Rubrics force an organization to focus on what is most worth measuring rather than what is easiest to count.

6

Rubric framework

A

B

C

D

Highly Effective
Effective
Somewhat Effective
Ineffective

What matters most to you In what areas will you measure success
What would "effective" or "ineffective" service look like
What indicators would characterize each level

Sample Rubric

(Activity Ten-Week Exhibit)

Quality of Exhibit

Active Visitor Participation

Attendance

Highly Effective

Memorable or evocative to visitors (75% or more of visitors can identify a "favorite" or "most interesting" work during exit interviews)
Receives consistently high critical acclaim

80% or more of visitors used one or more interpretive tools (e.g. the audio guide the children s treasure hunt interactive exhibit features or the exhibit catalog)

Weekly admissions reached target numbers in all 10 weeks of the exhibit

Effective
Somewhat Effective

60% of visitors identify a "favorite" or "most interesting" work
Most critical reviews favorable
50% of visitors identify a "favorite" or "most interesting" work
Mixed critical reviews

70% of visitors used tools 60% of visitors used tools

Weekly admissions reached target numbers in 8-9 weeks
Weekly admissions reached target numbers in 6-7 weeks

Ineffective

40% or fewer visitors identify a "favorite" or "most interesting" work
Critical reviews consistently negative

50% or fewer of visitors used tools

Weekly admissions reached target numbers in fewer than 6 weeks

7

Numerous cultural groups funded by the Geraldine R. Dodge Foundation in New Jersey have developed rubrics that not only help them describe their achievements to the foundation but more importantly enhance their ability to visualize success and manage their work to achieve their project goals. These New Jersey arts organizations developed rubrics to evaluate all dimensions of their work student learning a performance or exhibition marketing practices communications board and community engagement... even office cleanliness

How many columns or rows your rubric contains is not terribly important. What does matter however is choosing factors that are meaningful to your work and mission. Be as specific as possible about what you believe constitutes excellence and challenge yourself to define what data or indicators will be associated with each performance level.

Logic models are another powerful tool that arts organizations

harness for planning and evaluation. First popularized in the

nonprofit sector by the United Way in the 1990s logic models today

offer a blueprint that helps many arts organizations articulate their

desired outcomes and identify the inputs activities and outputs

necessary to achieve those goals. Logic models are especially useful

because they encourage "planning backwards." By first describing

your outcome goals then describing the resources and activities you will

Comedy Tragedy Fear K.G. McIntosh Georgia State Art Collection

pursue to achieve those ends logic models help to keep your work on track.

This approach also helps your evaluation stay tightly tuned to your program goals. Clarity and consensus

about outcome goals is vital. If your outcomes are vague or have different meanings to different people it will

be very difficult to determine what evaluation strategies and data will be productive.

Logic Model Framework
Outcome Goals What do we hope will change (for our constituents community or
organization) as a result of our work

Inputs
What people partners funds materials or space will we need to achieve our outcomes

Activities
What events action steps or activities will happen during our program

Outputs
What does our program produce
Attendees materials distributed memberships renewed artworks created artist fees paid funds raised etc.

Outcomes
What actually changed for our constituents community or organization How does that inform future goals

What indicators or data should we monitor throughout the process What data collection techniques will we employ to monitor those indicators

8

Sample Logic Model

Outcome Goal To improve the health and well-being of residents in the Senior
Living Center through dance and creative movement.

Inputs

Activities

Outputs

Outcomes

Two trained teaching
artists Creative movement
plan (approved by
Center physician) Support team of
Center nursing staff Space (recreation
room reserved 3
times weekly for 3
months) Artist fees ($200 per
artist per session)

Planning meetings
with coordinating
physician and staff
support team Promotion of program
to patients families
and all Center staff Artist-led movement
sessions (45 minutes
3 times weekly for 3
months) Participant interviews Staff medical
assessments (pre- and
post-program) of
clients Staff interviews

Number of
participating
clients Number of clients
participating in 8
or more sessions Number of Center
staff participating
or observing Interview reports Staff reports Artist observation
logs and journals

Clients actively
participate within their
abilities Participants interact with
each other artists and
staff Participants say they
enjoy the program Participants exhibit
positive physical effects
(increased range of
motion better
circulation stimulated
appetite more restful
sleep) Staff perceives
participation to be
beneficial to clients Artists find the
experience rewarding

Advice for the
beginner
Design your evaluation at the start of the project or funding cycle. Last-minute evaluations miss abundant opportunities to learn interact with constituents and collect useful real-time feedback throughout the course of a project. Combining evaluation with your strategic planning maximizes the benefits of both.
Start small. While comprehensive evaluation or large-scale research studies are sometimes appropriate even modest evaluation efforts can reap significant rewards. Start by evaluating one program or project.

GETTING STARTED
1. Articulate your outcome goals. What are you trying to learn change or accomplish with this program
2. Identify indicators. What kinds of information might reveal your progress or challenges
3. Inventory what you already know. Is there useful information already in place Does your experience suggest certain factors would be worthwhile to investigate
4. Get ready to secure additional information. What surveys interviews tests or other activities will need to take place
5. Brainstorm how you will use the information. With whom should you share what information and why How do those factors shape your evaluation plan
6. Ask for help. What staff volunteers or consultants will you need to assist with data collection analysis or reporting
9

Build a data "mosaic" that is unique to your program. Assemble a selection of indicators that are feasible to monitor and that can provide you with insight into your work and its effects. Quantitative and qualitative information complement one another so be certain to include some of each.
Do not be afraid to "do-it-yourself." Many kinds of evaluation can be conducted by your own staff board or volunteers. Taking a hands-on approach means that your evaluation will greatly benefit from the knowledge and experience (of your programs your audiences and your organization) that you bring to the task.
However know when to ask for help. Professional assistance is useful when you need to employ special research methods or conduct advanced analysis. The design of questions and sampling methods used for polls and surveys often can benefit from outside expertise as can the facilitation of focus groups or other feedback forums. When evaluating sensitive issues using a neutral "third party" evaluation also may help you to remain objective and avoid the perception (or the reality ) of bias.
Do not try to guess what your funders want to know. First consider your needs. Decide what methods and metrics will enhance your own learning and program success. Then if you are not sure what additional information your funders need ask. Most will be looking for evidence of good planning and evaluation practices and will be highly receptive to the indicators that are authentic for you. Grant makers often need specific information on your audiences finances and economic impact so you may have to collect additional data to comply with these requirements. But many grantees find this data collection much less onerous once systematic evaluation and performance measurement are in place.
Evaluation takes time so plan accordingly. Reserve time on your calendar to discuss and design the evaluation train your data collectors gather information and analyze it.

Cultivating a learning culture

Ongoing evaluation can empower many different dimensions of your work program design audience development promotional strategies governance and fund-raising just to name a few. These benefits however rarely accrue to an organization if one person alone is involved in the evaluation.

Establish an evaluation team.
Encourage individual and group reflection.

Consider who in your organization might become part of an evaluation team. Staff members artists and teachers involved in a project are logical stakeholders. Board members volunteers and former program

Make evaluation an ongoing effort.
Share your results.

participants may also be of assistance. Engage your crew in

brainstorming about the challenges and opportunities they believe your program is facing. Then discuss how

additional information or insight might facilitate the program s success. An evaluation team also can assist with the collection and analysis of information.

Look for ways to encourage your whole staff and board to reflect on their work at regular intervals. Sometimes the brightest ideas and epiphanies occur not during formal evaluation projects but during routine staff and board meetings or during informal discussions of questions such as

10

How is this program going What are we doing very well What could we be doing better What are our hopes and aspirations for the future of this program How can we get there

Embedding such questions into your work on an ongoing basis helps your organization learn and grow over

n

time. It also helps to build cohesion and a unified vision for your future success.

Recommended resources

Singing Our Praises Case Studies in the Art of Evaluation A comprehensive how-to book for the arts. Includes case studies from cultural organizations as well as practical advice on designing an evaluation choosing indicators collecting information conducting surveys and working a logic model from start to finish. Written by Suzanne Callahan and commissioned by the Association of Performing Arts Presenters with funding from The Wallace Foundation. www.forthearts.org/publications/singing.shtml
Measuring Joy Baltimore Clayworks executive director tells how evaluation using logic models helped her organization to strengthen their programs allocate resources and tell the story of their impact on young lives. Originally published in the National Arts Stabilization Journal in 2000 this article is excerpted by the National Endowment for the Arts in "Lessons Learned A Planning Toolsite." www.nea.gov/grants/apply/out/joy.html
Outcome-Based Evaluation A Working Model for Arts Projects This guide for arts organizations describes an outcome-based evaluation process and includes examples from opera dance and the media arts. Published by the National Endowment for the Arts. www.nea.gov/grants/apply/out/index-out.html
Cultural Vitality in Communities Interpretation and Indicators This monograph introduces a definition of community cultural vitality that encompasses opportunities for arts participation actual participation by the public and systems of support for cultural activity. Includes suggested indicators and sources of data. Written by Maria Rosario Jackson Florence Kabwasa-Green and Joaquin Herranz. Published by the Urban Institute. www.urban.org/publications/311392.html
Geraldine R. Dodge Foundation Assessment Initiative This Web resource summarizes nine principles of meaningful measurement. Several sample rubrics are provided to illustrate key evaluation concepts. www.grdodge.org/learning/assessment/index.htm

11

Arts Education Project Designer s Toolbook Describes outcome measurement frameworks for arts education. Written by Michael Sikes and published by the Idaho Commission on the Arts. www.arts.idaho.gov/ae/tool.aspx
Program Planning and Evaluation Using Logic Models in Arts Programs for At-Risk Youth A profile of how logic models facilitate planning forming collaborations artist training and evaluation of program outcomes. Written by Steve Heulett and published by Americans for the Arts. www.americansforthearts.org/NAPD/files/9491/Using%20Logic%20Models.pdf
More than Measuring A long-term evaluation report that includes tips for designing evaluations in ways that build the capacity of communities to design and improve arts education programs for children and youth. www.bigthought.org/
acknowledgements
About the Artworks The images contained in this publication are used with permission from the Georgia State Art Collection which includes works of sculpture photography printmaking painting ceramics jewelry fiber and silver produced by Georgia artists. Managed by the Georgia Council for the Arts the collection contains more than 600 original works. The collection is cataloged on-line through a partnership with Georgia Public Broadcasting and includes biographical information for many of the artists. Also included are lesson plans (consistent with core curriculum standards) that help educators use the materials to teach classroom units in the visual arts language arts science and Georgia history. www.gpb.org/stateart/home.htm
About the Georgia Council for the Arts The Georgia Council for the Arts (GCA) is a state agency with the mission to encourage excellence in the arts support the arts many forms of expression and to ensure that the arts are available to all Georgians. The agency strives for statewide coverage in the support of arts programming through grants which are competitive applications for funding that are adjudicated by peers. The GCA charter also encourages the agency to provide services and programming that supports the arts artists and the arts organizations of Georgia. www.gaarts.org/home.asp
About the National Assembly of State Arts Agencies The National Assembly of State Arts Agencies (NASAA) is the membership association of the nation s 56 state and jurisdictional arts agencies. NASAA provides knowledge services representation and leadership programs that help state arts agencies fulfill their many citizen service roles. NASAA also serves as a clearinghouse for data and research about public funding and the arts. www.nasaa-arts.org
The work of the GCA and NASAA are supported and strengthened in many ways by funding and programming partnerships with the National Endowment for the Arts which believes that a great nation deserves great art.
12