Can you predict how several individuals will react to the same action?  Within the
time it took the individuals to travel to a program, they each have experienced a diverse
set of happenings or variables, such as home life, traffic, public transportation, hunger,
illness, socio-economic hardships or job related worries, to name a few.  How can we
control all these variables to accurately predict what our program will accomplish?  This is
the challenge faced in the social sciences, unlike evaluation which leads towards absolute
predictability and replication as in the physical and natural sciences (Levine, 1995).

 The human being is a very complex and complicated entity.  Society, the
environment in which we live, also contributes to the unreliability of predicting what will
or can occur.
 To further complicate the evaluation process and control of variables, Ted Ward
suggests a three sector model of educational modes (Ward, 1995).

  Ward's model suggests three distinct educational modes, which borrow from each
other as the situation warrants.  The author will not attempt to define each mode due to
the nature of this writing, but will discuss the nonformal educational (NFE) mode and
provide suggestions for evaluation of NFE programs.
 
Nonformal education borrows it's methodology widely, from both the formal and
the informal modes of education.  NFE is change-oriented, deliberate, planned, staffed and
financially supported.  It is functional, unrestricted as to time and place, usually responsive
to need and is an effective tool for development.  NFE encompasses a broad spectrum that
can range from a youth learn to swim program to an agriculture workshop concerning
pesticide application. "The term 'nonformal education' has provided a contrast; its major
use emphasize new and different change-oriented education.  Nonformal education usually
means a highly functional learning linked specifically to particular social interventions"
(Ward, 1995).

Why Evaluate ?

 The concept of evaluation is a very interesting and important discipline.
Educators, administrators, politicians, executives, funders and most of society practice
some level of evaluation in their every day lives.  Tasting a new food, pricing and
comparing the cost of several items, selecting a color paint for their residence are all
examples of  evaluations, limited as they may be.  Data  must be collected through the use
of designed instruments to formulate an analysis of the outcomes of any action, event or
program.  Exploration of the who, what, where, why, how much and how many questions
may occur.

 The aforementioned process of evaluation sounds elementary.  "The assumption is,
as an outcome of evaluation we will now be able to offer the program in the future with a
better chance of success" (Levine, 1995).  Levine also suggests that we do learn more
about the program evaluated, but we cannot predict with accuracy if the program will be
improved in the future.

 "It is essential that the [educator] be able to document change as a function of
[programming], using both formal and informal methods.  Crucial to the functioning of an
effective [educator] is the ability to design and use evaluative tools and methods to
enhance the potential of [programs].  Program evaluation allows the educator to facilitate
meaningful local change, while providing feedback" (Applied Research and Evaluation,
1995).

Know Your Audience

 A phrase which crosses all disciplines, "know your audience", is also appropriate in
evaluation.  Who requested the evaluation ?  What are the philosophies and values of the
organization providing the program and the one requesting/requiring the evaluation ?  Are
the findings of the evaluation to be presented in a formal or informal setting ?  What are
the requirements of the stakeholders of the evaluation - descriptive information or change-
oriented evidence ?  These are all questions which will assist in targeting the audience for
the evaluation.  Do not make the assumption of thinking the evaluation audience are the
participants of the program, they  seldom are.

  The ideal evaluation process should begin during the program development stage
as suggested by Dr. Claude Bennett, USDA, and Dr. Kay Rockwell, University of
Nebraska, Targeting Outcomes of Programs (TOP):  An Integrated Approach to
Planning and Evaluation recommends stakeholders of the program to be identified and
included in the planning, implementation and evaluative processes.  This forms a "team
approach" from vision to conclusion.  This process makes sense and should be adopted, if
appropriate.

 Dr. S. Joseph Levine, Professor,  Michigan State University, suggests the program
and evaluation should provide consistency in philosophy and methodology.  Therefore,
evaluative methodology and the instruments used are recommended to be included in the
program development plans and proposals.  This also is suggested in the TOP model as
well, to include the stakeholders throughout the total program planning, implementation
and evaluative processes.

 Due to personal and professional commitments, the ideal program planning
situation seldom occurs.  Therefore, the reality of  "who and how will these outcomes be
utilized" must be explored.  If the answers are, "I don't know", "no one" or "the results
will be used at a later date to prove accountability", one must question the use of valuable
time spent to develop, implement and analyze the program.  Conversely, "with today's
tightening budgets, an increasing concerned 'public consumer' wants to be assured that the
expenditures of time and money resources will result in a fair return on an investment of
tax dollars and private contributions" (Andrews, 1981).  One must then decide which
programs need evaluated and to what extent, such as, securing descriptive data or
conducting a more endepth evaluation to provide evidence of  knowledge gained or
practice changes.

Are there evaluative models ?

 Dr. Robert Reineke, Associate Professor, University of Nebraska, presented an
overview from Worthen and Sanders text, Educational Evaluation: Alternate Approaches
and Practical Guidelines, as a presentation to a Michigan State University graduate class,
Program Evaluation in Adult Education.  Six suggested general evaluation
models/frameworks were discussed.  The review will follow.  Please reflect on Dr.
Levine's suggestion, there must be consistency between the philosophy of the program and
the methodology of the evaluation.

Six General Evaluation Models/Frameworks:

Objectives Oriented:  where the focus is on specifying goals and objectives and
 determining the extent to which they have been attained.

Management Oriented:  where the central concern is on identifying and meeting the
 informational needs of managerial decision-makers.

Consumer Oriented:  where the central issue is developing evaluative information on
 educational "products," broadly defined, for use by educational consumers in
 choosing among competing curricula, instructional products, and the like.

Expertise-Oriented:  which depend primarily on the direct application of professional
 expertise to judge the quality of educational endeavors.

Adversary-oriented:  where planned opposition in points of view of different evaluators
 (pro and con) is the central focus of the evaluation.

Naturalistic and participant-oriented:  where naturalistic inquiry and involvement of
 participants (stakeholders in that which is evaluated) are central in determining
 values, criteria, needs and data for the evaluation (Reineke, 1995).

When to evaluate ?
 
Evaluation is an ongoing process which, as stated previously, should be included in
all program development stages.  The real question is when to start collecting evidence, or
data, to use for the evaluation.  Research and Development in Global Studies suggests
there are six phases in a project when it is appropriate to consider collecting evidence for
evaluative purposes:

Project Design:  The most frequent form of evaluation at this point is a needs assessment.
 Program directions can be influenced by the needs of the participants.

Resource Development:  This involves testing learning materials and delivery strategies
 before the start of the program.

Program start-up:  To establish a baseline to judge changes in knowledge, behavior or
 practices of the participants.  Another reason is to assess whether resources are
 being deployed properly.

In-progress:  To monitor the actual operations in case adjustments are required and to
 prepare for summative evaluation.

Program wrap-up:  Explores end of program reactions and if objectives were adequately
 met.

Follow-up:  The period after the program has ended.  This technique may suggest long
 term benefits of the program as evidenced by practice and behavior changes.
 (Research and Development in Global Studies, 1988).
 
 

Stages of evaluation.

 Planning the stages or direction of the evaluation is as important as when to
evaluate.  The Research and Development in Global Studies organization also suggests a
reasonable way to plan an evaluation:
 

•  Setting aims:  What aspects of our project need evaluating?
•  Launching the evaluation:  Who will do it and in what parameters?
•  Constructing the instrument:  How can we prepare the instruments we require?
•  Collecting data:  How can we obtain the information we require?
•  Analyzing results:  What does the information tell us?
•  Delivering the pay-off:  How do we present and make use of the results?

 A minimum of three unique phenomena which may occur in the NFE mode are
suggested:  praxis, or reflection and action, unanticipated learning and the development of
a  learner-centered environment.  These key elements of a NFE program should be
explored in order to provide evidence in the evaluative report to share their importance
and uniqueness if present.

 Paulo Friere's term praxis, suggests a "dynamic concept that consists of two major
ingredients-action and reflection" (Key Concepts of Nonformal Education, 1993).   In
essence, this means moving from thinking of what has been done and how to improve it,
to acting on your reflections and putting it to use as an action.  "The idea of praxis is so
very significant for nonformal learning.  It can be viewed as the essence of how learners
can continue to grow and develop through their own insights and actions" (Key Ideas and
Concepts of Nonformal Education, 1993).

 One could suggest an addition to Bennett's Hierarchy of Extension Programs.
Instead of looking at the design as a hierarchy, meaning each level must be
achieved before moving to the next, separate each level and add praxis, a time to reflect
upon each level which enables one to move forward or reverse direction when needed.
This reflective time also provides an opportunity to direct a well thought out action in the
appropriate direction as seen by the evaluative/program planning team.
 
  The two remaining terms, unanticipated learning and learner-centered are unique,
existing at times independently and at times concurrently.  Providing a learner-centered
environment produces the opportunity for meeting the needs of the learner because their
needs may be explored, recorded and measured accordingly by direct observation or
questioning.  "Unanticipated learning [is] problem [needs] focused.  Unanticipated
learning [is] immediate" (Levine, 1995).  Dr. Levine also suggests,
 

•  unanticipated learning usually occurs as a function of dialogue and interaction,
•  unplanned learning takes place on the way toward the clear educational goal,
•  unanticipated learning occurs at a very significant level of meaning for the learner,
•  unanticipated learning usually occurs in response to a learning need that may have been lurking just below the surface for the learner.

 Praxis, unanticipated learning and a learner-centered environment are key elements
in a NFE program.  These practices also provide the unique and change-oriented
environment which Ted Ward described as conducive for nonformal education to occur.
Therefore, plan to include these unique experiences in your collection of data as part of
the evaluative process of NFE programs.

 Be sure the program philosophy is consistent with the methodology chosen for
evaluation.  Know who will use the information collected and presented and most of all be
prepared to share the results of your evaluation.
 

Bibliography

Andrews, Dr. Mary and Thompson, Christine. 1981. "The Evaluation Pipeline,
 Accountability Reporting," Michigan Cooperative Extension Service, Michigan
 State University.

Bennett, Claude and Rockwell, Kay. 1994. "Targeting Outcomes of Programs (TOP):  An
 Integrated Approach to Planning and Evaluation," United States Department Of
 Agriculture.

Case, Roland, Andrews, Mary and Werner, Walter . 1988.   How Can We Do It?  An
 Evaluation Training Package for Development Educators.  Michigan State
 University, East Lansing.

Levine, S. Joseph. 1995. "Catching a Breath of Fresh Air:  The Challenges Facing The
 Extension Educator," Department of Agriculture and Extension Education,
 Michigan State University.

Reineke, Robert. 1995. "Worthen & Sanders:  Six General Evaluation
 Models/Frameworks,"  Program Evaluation in Adult Education and Training, AEE
 891, Michigan State University.

Ward, Ted (a).   "Nonformal Education--What Is It?'',  1995 Nonformal Education
 Institute, Department of Agriculture and Extension Education, Michigan State
 University.

Ward, Ted (b).  "Two Modes of Nonformal Education," 1995 Nonformal Education
 Institute, Department of Agriculture and Extension Education, Michigan State
 University.

__________. 1995. "Applied Research and Evaluation," Core Competency:   Applied
 Research and Evaluation, Michigan State University Extension.

__________. 1993. "Key Ideas and Concepts of Nonformal Learning," Department of
 Agriculture and Extension Education, Michigan State University.
 

Top

Back to data collection instrument list