|
-
-
|
|
Link to printable page
 Evaluation Techniques
A variety of evaluation techniques has been employed as the demonstration sites have implemented their evaluation plans. Some are qualitative, some are quantitative; some are direct while others are indirect. Some deal with process, others focus on outcomes. Some yield descriptive data, others involve analytical methodology. Overall, the project has chosen a "soft path" evaluation approach to enable relationships with the Latino community to develop without intrusive or high-level scrutiny.
Data collection has included many of the usual activities—such as keeping attendance records, plotting participation patterns, logging home visits, and tracking hours of community service. Other data have been generated from end-of-session evaluation forms (for example, at the conclusion of a training workshop for youth mentors and leaders and at end of the school year for an after-school program) and pre- and post tests (for youth in the Video and Media Arts club and for families enrolled in the family computer learning program). In addition, the project has also adapted some familiar techniques and tried some new approaches to capture process and outcome information and accommodate cultural factors. Some of these are described below.
Needs and Assets Assessment
In the needs/assets phase of program development, two demonstration sites organized community-based focus groups to help determine program direction. Participating in the groups were Latino youth and parents, school officials, and representatives of various community organizations and agencies with interests in the Latino population. A third site formed a collaboration of youth- and family-serving agencies with ties to Latino clientele, and through their work with that collaboration identified a programming niche.
Hanging Out
One of the sites placed a program assistant (a young Latina known and
respected in the community) in local high schools on a revolving basis.
She "hung out" (with the school's permission) for weeks in
cafeterias, libraries, and other gathering spots, becoming acquainted
with Latino students, and talking with them to identify and test ideas
for programs. Their ideas led to the development of the Tech
Wizards, which have been extremely popular. (Some see this technique
similar to the "field notes" approach often used in anthropological
research.)
Visualization
Another demonstration site used what was described as a "visual variation of a focus group." Staff brought together a group of Latino students and parents, school officials, and Extension staff. Some were bilingual, others monolingual. Through translators, group members were asked to visualize a future for their children and create a simple poster or collage to communicate that vision. Paper, paste, tape, paints, pens, scissors, magazines for clipping, and other supplies were available. The activity served as an icebreaker, helped to identify program direction, and began to build rapport and a support base as people realized the commonalities of their visions.
Focus Group Study
At the outset of the project, focus groups were used to increase understanding of the Latino culture as it relates to volunteerism in community-based organizations. Three focus groups involved a total of 18 participants, including 13 who were Latino. All had actively recruited and supported Latino volunteers in their organizations. Transcriptions of taped discussions formed the basis of a qualitative study, with data analyzed and interpreted using a schematic content analysis strategy. Findings from the study were extremely useful and have been applied in the project to develop strategies for recruiting, supporting, and recognizing volunteers. They have also been published in a widely distributed publication, Recruiting and Supporting Latino Volunteers.
Dialogue and Reflection
In the spirit of The Reflective Practitioner (Schön, D.A., Basic Books, 1999) and Evaluative Inquiry for Learning in Organizations (Preskill, H. & Torres, R.T., Sage Publications, 1999), dialogue and reflection are used by state and local staff in annual program reviews and occasional meetings to check program progress. The resulting data have been framed in terms of "successful practices" that address building partnerships and relationships with the Latino community, developing culturally appropriate programs for youth, and staffing factors related to paid and volunteer staff. Findings have also been incorporated into an informational table relating to Latino cultural factors and their implications for Extension programming.
Trolling and Eavesdropping
These two related techniques take advantage of the indirect communication patterns characterizing Latino culture. Project staff circulate during an activity (a family program, for example) alerted to listen for comments they are intended to overhear. A variation, similar to a "listening post" technique, involves orienting designated listeners to troll a room or activity. These designates listen for comments relating to particular aspects of programming or ask previously determined questions to individuals or small groups of participants in an informal way.
Data Collected as Part of an Activity
Examples here include periodic end-of-activity question-and-answer sessions to review learning and check comprehension, de-briefing of field trips or other special activities, and incorporating assessment opportunities into the ongoing learning experience. For example, youth in the Video and Media Arts project were videotaped at the beginning of the project answering, "What do you expect from this project?" and at the end of the project responding to "What did you get out of this project?" At the same time they were learning skills in videotaping (at the beginning) and demonstrating taping and production skills (at the end). Another example is setting up opportunities for youth in a computer class to individually retrieve and respond to e-mail messages sent by the instructor, to determine whether they are able to perform this task.
Program Reviews
The program review is a participatory process where state staff and local project teams meet collaboratively for an in-depth look at a local program. It is intended to be a congenial, informative session where local staff can showcase the local program, while at the same time state staff can become more knowledgeable about program activities and more aware of how they can best support local efforts. The reviews occur annually at each local site and are timed to take advantage of such regularly scheduled reflective activities as annual planning and reporting.
Questions for state staff to pursue are selected from a set of items addressing each major element expected to receive attention in local programs, including educational programs, collaborative efforts, involvement of volunteers, connectivity, and sustainability. Some questions may not be appropriate for some sites; additional questions can be added to the list or spontaneously asked as the discussion progresses. The overall tone of the session is conversational rather than interrogative, and it is expected that questions and responses will generate productive discussion.
A schedule of review activities is jointly prepared in advance of the review, with input from both state and local staff. A description of the overall process and a list of review questions to be specifically addressed are shared with local staff before the site visit. Local staff members are expected to provide any additional information, materials, tours, and so on, they feel would communicate information about their program. They are also invited to formulate their own set of questions for the state staff. The final agenda item involves a summary of observations, reactions, recommendations, and expectations expressed by both groups.
Benefits and Accomplishments List
The Benefits and Accomplishments List captures and summarizes project
outcomes other than those related to educational objectives—such
as expanded resources in state and county offices, project experiences
shared with others, recognition of project staff and youth expertise,
and organizational change impacted by the project. The list provides
a cumulative, statewide perspective of the total Oregon Outreach program
across project sites as well as at the state level.
For an example of this list, see Benefits
and Accomplishments, Oregon Outreach State-Strengthening Project.
|
|
