Evaluation and Program Planning 25 (2002) 1±14

www.elsevier.com/locate/evalprogplan

Participatory program planning: including program participants

and evaluators
Laura Nichols*
Department of Anthropology/Sociology, Santa Clara University, 500 El Camino Real, Santa Clara, CA 95053-0261, USA

Abstract
Many evaluators have written about the usefulness of including stakeholders and program participants in the design and implementation of
program evaluations. Policy makers are also beginning to see bene®ts of including those most affected by policies in their work. However, there
has been little attention paid to program planning and how future program participants can assist in the process. Because of their knowledge of
program design and their experience with participatory evaluations, many evaluators are well equipped to conduct participatory program
planning. Including future program participants and evaluators in program planning could not only improve programs, but also allow for
evaluative measures to be included and utilized in the design and administration of new programs. Presented in this paper is a generalizable
program planning model that includes participants in eight planning components. Some of the pitfalls of participatory work as well as examples
of how participatory planning could be used in various contexts are provided. q 2002 Elsevier Science Ltd. All rights reserved.
Keywords: Program planning; Participatory planning; Program evaluation

1. Introduction keeping and dif®culty gaining access to data can present

many obstacles for evaluators.
For almost 20 years evaluators have been writing about Many of these problems could be eliminated if addressed
the bene®ts and problems of including program participants when programs are being planned in a climate where stake-
and other key stakeholders in the design and administration holders and participants see each other as partners in ensur-
of program evaluations (see for example Ayers, 1987; Folk- ing program success. Yet compared to evaluation and policy
man & Rai, 1997; Greene, 1987; Mathie & Greene, 1997; research, little has been written about how to plan programs
O'Sullivan & O'Sullivan, 1998; Weiss, 1983) and policy as well as how participants can be part of the process.
making (Doherty & Rispel, 1995; Rixecker, 1994). This paper attempts to overcome these shortcomings with a
Although challenging, researchers have noted the impor- generalizable program planning model that includes compo-
tance of such inclusion not only in improving evaluations nents which: (1) emphasize program design areas that
and the utilization of evaluations, but also in empowering program evaluators often note as weak when conducting
participants (Gergen & Gergen, 1991; Harris, 1995; evaluations (2) include explicit opportunities to incorporate
Morgan, 1983; Papineau & Kiely, 1996; Reason, 1988). potential program participants and other program stake-
One of the major dif®culties in conducting participatory holders in the planning process and (3) are cognizant of the
evaluations is overcoming the power differences ingrained larger context within which programs are planned. Although
in a service model that often disempowers program partici- the speci®cs of component implementation may depend on
pants. Dealing with the con¯icts that often occur when the program being implemented and the composition of the
working with heterogenous groups often requires much planning group, this planning model includes a description of
time and attention by evaluators (Greene, 2000; Mathie & each component and how participants can be included in that
Greene, 1997). Further, trying to determine the goals and component. The challenges and potential pitfalls inherent in
theory underlying a program and balancing the multiple each component are also discussed.
viewpoints of what is considered program `success' is
often a challenge. This combined with inadequate record
2. Program planning

* Tel.: 11-408-551-7131; fax: 11-408-554-4189. Since the end of World War II, interventions in the form
E-mail address: lnichols@scu.edu (L. Nichols). of programs have been the approach used to address social
0149-7189/02/$ - see front matter q 2002 Elsevier Science Ltd. All rights reserved.
PII: S 0149-718 9(01)00044-1
2 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14

problems and human needs in the United States. The typical understanding of the structure of the program and the
procedure has been to identify the problem and design a stressors faced by program administrators and staff ulti-
program intended to solve it using private and public mately reducing con¯ict. Participatory program planning
support (Hodgson, 1978). Yet the problems have not disap- also has the potential to strengthen the evaluative potential
peared and some program evaluators have bemoaned that of programs. Involved in the planning stages, evaluators and
ªprograms don't workº (Martinson, 1974). Program design participants can incorporate evaluation measures into the
has generally been left up to policy makers or, most often, implementation and administration of the program. Finally,
individual program administrators. A systematic approach a participatory planning approach provides a strong founda-
to the development of social programs has generally not tion from which programs can incorporate participatory
occurred, perhaps because of the diversity of needs that decision-making and participatory evaluations into the
programs are designed to address. However, there have everyday workings of the program (Mathie & Greene,
been some attempts to create generalizable program plan- 1997).
ning models. Two of the more widely known models are Participatory research is not easy. Program participants
effectiveness-based program planning introduced by Kett- sometimes live a very different reality from paid program
ner, Moroney, and Martin (1990) and the rational planning staff and other stakeholders. In participatory evaluations, the
process designed by Mayer (1985). power differences between participants and others on the
Kettner, Moroney, and Martin (1990) discuss program evaluation team as well as the challenge of dealing with a
planning in terms of inputs, throughputs, and outputs. Inputs very heterogeneous group have added time to the process
include participants (whom the authors refer to as clients), and put added strains on evaluators (Kelly & Van Vlaende-
staff, material resources, facilities, and equipment. The ren, 1995; Mathie & Greene, 1997; Palumbo & Hallett,
throughputs refer to the service delivery process including 1993). Small group research tells us that to overcome differ-
the activities which make up the services provided. Outputs ences and work together well, groups with members who are
involve the de®nition of the individual termination point for unequal must meet the following three conditions: (1)
each participant and the ®nal individual outcome. members must have sustained contact over time; (2) they
Mayer's (1985) model, on the other hand, is more abstract. must relate to one another based on equal status; and (3) all
He de®nes program planning in terms of a rationality strategy members must agree to follow the same rules of interaction
in which ef®ciency, optimality, and synthesis are assessed. (Cook, 1988). The planning process presented in this paper
Mayer considers the importance of the organizational context provides this context. It is up to the evaluator (also referred
in which the planning occurs, whether it be done by an inde- to as the facilitator) to make sure that these three conditions
pendent body, the program administrators, or a participatory are met and upheld throughout the process.
model in which citizens impacted by the program are The participatory program planning model depicted in
involved. While Mayer discusses the importance of involving Fig. 1 includes components that provide participatory condi-
persons who are served by programs in the planning process, tions at the same time as they address some of the weak-
he does not put them at the center. nesses evaluators often uncover with program design. Each
planning component not only allows for the collection of
important information that could strengthen new programs,
3. Participatory program planning but does so in a manner that utilize the knowledge and skills
of participants. The planning process is led by a planning
Similar to participatory evaluations that emphasize the group made up of potential participants and other stake-
active inclusion of program clients, participatory program holders. Although not illustrated in the model, evaluators
planning puts participants at the center of the planning are envisioned as the facilitators of the whole planning
process. Participatory work can help to empower partici- process.
pants 1 as well as teach participants (and other members of
the planning team) valuable decision-making, communica- 3.1. Role of evaluators
tion, research, and other skills while changing the structure
of the program from a top±down service model to a colla- Evaluators who conduct participatory and/or stakeholder-
borative service approach. based evaluations are particularly well trained to assist in a
Including the viewpoints and experiences of future participatory program planning process. Using a variety of
program participants when planning programs not only methodologies, evaluators typically know how to include a
bene®ts the program participants, but also the program. number of people in a research process.
Programs that ®t individuals' needs, wants, and experiences In participatory program planning evaluators serve as
will have greater success at attracting and keeping partici- facilitators to the process or as partners (as encouraged by
pants. Through participation, participants have a greater Mathison, 1994). Facilitators with negotiation, group facil-
itation, listening, interviewing, and training skills (Ayers,
1
For a summary of current literature on the role that participatory evalua- 1987) and who are seen as trustworthy to all members of
tion can play in empowering participants see Papineau and Kiely (1996). the planning group will likely be the most successful in
 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14 3

Definition of
Needs & Program
Program Theory
Purpose

Input of
Program
Participants

Resource and Program

Asset Mapping Goals

Program
Stakeholder Implementation
Check-In

Policy
Ecological Considerations
Environment
Assessment

Evaluation
Program Plan
Design or
Replication

Fig. 1. Participatory program planning components.

leading a participatory planning process (Folkman & Rai, evaluation of the program might result. To mitigate this
1997). Facilitators must also be committed to and convinced potential problem, after the program is implemented, the
of the importance of including program participants in the use of an external evaluator is suggested.
planning process and be creative in determining how best to
use the assets of all members of the planning group. 3.2. Program participants
Throughout the development of the program the planning
group needs consistent evidence that the facilitator is lead- As discussed previously, the input of the persons who will
ing them through a useful and legitimate planning process. be utilizing the program can often be overlooked. Yet the
If members of the group cannot see how each planning involvement of potential program participants could provide
component is contributing to the development of the vital information that will help to ensure the success of the
program, they will be less willing to participate in future program. ªThe insights of persons who experience the
planning components. If facilitators are unsure how a parti- condition in question often transcend the boundaries of
cular component could be incorporated into the planning the formalized knowledge of the professionalsº (Mayer,
process then they need to decide if the group could help 1985, p. 150). This knowledge is especially important in
with this determination or if they should not use this compo- determining if a particular program will indeed be success-
nent. It is important that evaluators are given the power and ful and meet the needs of a speci®c population. How often
authority to direct the process and that outside groups or do we ask failing (or succeeding) students what is necessary
individuals will not disregard the work of the group for them to learn? Ask homeless people what is needed to
(Greene, 2000). end homelessness? Individuals experiencing such circum-
Because of evaluators' extensive involvement in the plan- stances can shed new insight into the structural dimensions
ning process, the evaluator's relationship to the program is of problems and potential solutions that may not have been
similar to that of an internal evaluator (Ryan & Johnson, considered by program planners, policy makers, or social
2000; Weiss, 1972). If the closeness of this role is not scientists.
recognized, confusion, con¯ict, and later bias in the ®nal The inclusion of target populations can do more than
4 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14

inform research; their involvement also allows program over time may ®nd the participatory planning process more
participants to become actively involved in addressing useful than those who serve people on a one-time or very
their situation. A participatory action research approach limited basis.
provides individuals with the power to control their lives, There are a number of ways that participants can be
learn new skills, and ultimately create a situation where included in the program planning process. Three relatively
researchers are planning with, rather than planning for, straightforward ways are: (1) putting together a small group
target populations (Swantz & Vainio-Mattila, 1988). After of future program participants to serve as members of the
program implementation, current and former program parti- planning group (2) talking with groups of potential program
cipants can guide program improvements based on the participants who have a particular status or characteristic,
results of participatory evaluations. and (3) querying the whole population or a sample of parti-
Including program participants, especially those who cipants (through focus groups, interviews, surveys, etc.). At
have been marginalized in society, presents a number of a minimum, to be participatory, some participants must be
issues and opportunities. While participatory approaches part of the planning group and be seen by others in the group
call for the views of program participants to be included, as active and crucial members of the planning process.
their viewpoints are likely in¯uenced by their position as Planning group members must be made aware that they
disempowered individuals (Risman, 1993). Varying degrees are serving as representatives of a larger body of potential
of consciousness combined with situations where partici- program participants and must therefore agree to not only
pants and stakeholders hold different amounts of power focus on their own needs but also the potential needs and
can make it dif®cult to work towards any collaborative interests of that larger group. When possible and feasible,
approach leading to change (Collins, 1990). ªOne of the input from that larger group of program participants can be
gravest obstacles to the achievement of (social change) is sought.
that oppressive reality absorbs those within it and thereby In a situation with adequate planning support, program
acts to submerge human beings' consciousnessº (Freire, participants could be paid and program participants and
1970, p. 33). stakeholders would be interviewed before becoming a
To overcome this potential problem, the planning process member of the planning group. All group members must
must allow participants to re¯ect on their lives and the many also agree to treat other members of the planning group as
factors that may have contributed to the problem that is to be equals and must follow agreed upon rules for discussion and
addressed by the program. The planning process must also decision making (Ryan & Johnson, 2000).
allow for opportunities for consciousness raising among
participants and stakeholders. Thus, the planning process 3.3. Stakeholder check-in
itself provides participatory dimensions that allow partici-
pants to be empowered and brings together members of the For a program to have a practical and positive impact, it is
planning group. necessary that program planners and future program parti-
Conditions for participation. For participants to be effec- cipants work together to devise a program which will serve
tively included in the planning process there must be a the most crucial needs of program participants. This
program environment where planners and key stakeholders requires consideration of important stakeholders and their
are receptive to inclusionary practices (Papineau & Kiely, multiple agendas (Hallett & Rogers, 1994). Stakeholders
1996). Facilitators must also understand the ability that can include, but are not limited to: future program admin-
participants have to evaluate their situation and make istrators and staff, political of®cials with a stake in the
programmatic suggestions. The potential problems of inclu- program, community leaders, and academicians.
sion can be addressed at many stages of the program Checking in with stakeholders can take a variety of forms.
planning process. Program participants and other members In some cases the evaluator serves as the liaison between
of the planning group might require training in some areas program participants and other stakeholders. Yet Guba and
of non-pro®t management, leadership, and effective Lincoln (1989) discourage that model and instead encou-
communication skills. Learning a variety of skills will not rage that stakeholders and program participants be brought
only assist in the program planning process, but also can together to form consensus as a group. They argue that
serve as tools of training and empowerment for participants including everyone in one group will help diminish the
that can be helpful throughout their lives. unequal representation of disadvantaged persons in the
One of the most important parts of participatory program normal evaluation process. Facilitators may wish to follow
planning is de®ning who are potential program participants. this approach to prevent being put in the middle of a poten-
Key stakeholders can help locate potential participants by tially divisive situation; if participants and stakeholders
introducing the evaluator to individuals who are experien- have no contact during the planning process an `us' versus
cing the situation that the program is hoping to address. `them' dynamic may develop which could severely harm the
These individuals can add further insight as to who else planning process and the future success of the program.
should be included in the planning process. Programs True or collaborative consensus between all stakeholders
designed to have frequent contact with the same participants and participants has been criticized for being idealistic and
 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14 5

impossible to achieve (Fishman, 1992; Palumbo & Hallett, education program on a college campus that targets women
1993). And great differences between all those involved in students in the residence halls, 2 the second is the develop-
the process could impede implementation (Mathie & ment of the programming for single parents and their chil-
Greene, 1997). A planning strategy that provides opportu- dren living in transitional housing.
nities for stakeholders and participants to come together
early in the process can help provide a context from
4.1. De®nition of needs and program purpose
which collaborative consensus can occur. However, if stake-
holders are not truly committed to participatory planning Recognition of the purpose of a program is an important
and the facilitator thinks that they might impede participants ®rst step in designing any type of program. Many times
from fully participating in the process, then separate meet- programs are developed to address a particular social
ings, with the evaluator as the liaison, may be appropriate. problem or need. In this case, determining the root cause
The planning components presented next provide a colla- of a problem is critical to planning an appropriate interven-
borative context at the same time that they allow for the tion (Kettner & Daley, 1988).
collection of important information which can be utilized It is crucial that the problems and their causes be de®ned
to design an effective program. Through balanced and pro- not just by policy and claims makers, but also by the persons
active participation, and with the help of facilitators experi- most impacted. These problems can then be translated into
enced in bringing together people with varying interests, needs and program purposes in ways that make sense to
this process allows for more resources and ideas to be potential program participants. Hopefully this will minimize
brought to programs, ultimately resulting in the develop- the chance that the program will fail due to lack of use.
ment of better interventions. Often the reason for program failure is due to the fact that
planners did not take into account the situations of the
persons using the program, and had little to do with the
4. Program planning components actual `nuts and bolts' of the program (Swantz & Vainio-
Mattila, 1988).
There are many problems with program design that are Opportunities for participation. Program participants can
often discovered in the course of a program evaluation. be actively involved in both giving their perspectives on the
Program evaluators often note that an unclear de®nition of major issues that need to be addressed in their lives or
a program's purpose, goals, theory, and inadequate baseline communities as well as ®nding or collecting any data that
measures often make it dif®cult to adequately design and might exist on the extent of the need. Planning group
implement effective program evaluations (Posavac & members can brainstorm issues, survey or do a focus
Carey, 1997). Programs are often designed in a vacuum, group with the program population on their concerns, and
forgetting the larger community and policy context that, if ®nd existing studies on challenges the population often
taken into consideration, could provide important assets for faces. For example, the planning group for the peer educa-
the program. Further, when evaluators are attempting to tion program conducted focus groups with resident advisors
conduct participatory evaluations they often have problems at the same time as they found existing data on issues often
bringing various stakeholders together who have never had faced by female students. A skills training program for
to work together in the past. adults could interview potential participants. The results
Each of the remaining program planning components of these inquires are then brought to the whole planning
included in Fig. 1 provide participatory strategies that group and used to determine the purpose of the program.
address these problems during program planning. The The de®nition of needs is an important start to participa-
purpose of each component, its relationship with other tory planning because it is one of the ®rst opportunities for
components, how participants can be involved in that evaluators to lead participants and stakeholders through a
component, and some of the potential pitfalls of that plan- process where they learn the many facets of a problem. As a
ning component are discussed next. Although the compo- result of this component, planning group members should be
nents are presented in an order that may prove useful to able to see how structural factors contribute to problems
follow in the planning process, components can be (Baca Zinn, 1989). Program participants and stakeholders
conducted simultaneously or in varying sequence. This learn early on that individuals are not solely at fault for their
planning model can be adapted to a variety of programs situations. This may help to empower participants and
particularly in the human service and education realms. provide an opportunity for stakeholders to better appreciate
While this paper is limited to a theoretical discussion of the experience and circumstances of program participants.
the planning process, Table 1 in Appendix A may be useful Engagement in de®ning the needs or challenges as well as
to readers as it provides a summary of the major planning the purpose of the program also allows for the planning
components as well as an illustration of how each of these group to understand up-front the limits of the program and
components could be incorporated, in a participatory
manner, in the development of two different programs in 2
For more speci®c information on this program and the outcomes of
two different contexts. The ®rst is the planning of a peer some of the participatory components see Nichols and Lumley (1999).
6 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14

to consider how to deal with other needs or challenges that the program and/or issue being addressed by the program.
may be future barriers to the program's success. The survey should be general enough so other skills could
Potential pitfalls. The most dif®cult part of implementing be added later. Then, through interviewing every program
this component is in deciding what part or parts of the needs participant, the skills of each could be determined (skills
or issues the future program should address. Each stake- such as ®ling, playing an instrument, being bilingual,
holder and participant involved in the planning process etc.). In the ®nal step, the planning group can decide how
may have a different idea. Working out these differences to best utilize those skills in the program. Information can
and hidden agendas will take time and perhaps multiple also be collected on those types of skills that program parti-
meetings. However, bringing out these differences early in cipants would like to learn and participants can be matched
the planning process will help put the planning group on based on skills and wants. For example, participants with
similar footing and make the purpose of the program clear. computer, language, or music skills might be able to tutor
It is not necessary for there to be complete agreement on the other program participants or teach or assist with classes the
needs and the purpose of the program at this stage as program offers. Participant skills and contributions are
succeeding planning components will help to better de®ne de®ned based on their interests and capabilities. A devel-
and re®ne these ideas. However, at this point facilitators opmentally disabled participant with artistic skills could be
should be careful to be up-front with participants about responsible for managing the program's art supplies and
any limitations or parameters that are being put on the plan- decorating name tags for other participants. The point is
ning process and/or the program being developed. that each participant is seen as having assets to offer the
program and these skills are utilized by the program in
4.2. Resource and asset mapping various ways. The survey of skills could also be used at
intake after the program is implemented to continually
Resource and asset mapping provides the opportunity to utilize the assets of new program participants.
determine the skills and assets that participants and the A similar process could be used to determine the
larger community bring to the planning process and the resources and assets available in the larger community.
program being planned. There are very speci®c participa- Participants can be involved in researching other commu-
tory techniques that can be used for this purpose. nity agencies and organizations as well as potential funding
McKnight and Kretzmann (1996) provide the most sources.
complete explanation and tangible example of how to deter- Potential pitfalls. At this point it may become clear that
mine assets and resources in communities. Their model can some stakeholders involved in the planning process do not
be transferred to the program context. They emphasize the see program participants as truly people with assets to share.
assessment not only of needs and problems but also assets Facilitators should watch for stakeholders who try to limit
and resources at three different levels. In a program context the involvement of participants in the planning process and
this would start with assets located in and controlled by the program.
program participants. Next is a consideration of assets to Further, because the resources and assets individuals
the program that are outside the control of participants, bring to a program may change with time and the turnover
and the least accessible are assets located outside the of new participants, the program cannot be planned solely
program (such as funding sources or other programs). based on the skills of those participants involved in the
An inventory of the resources available to the program in planning process. Similarly, resources in the community
the community is an important step in determining what may change over time. Facilitators should caution planning
services already exist (Mayer, 1985) and preventing dupli- groups not to make unique skills foundational to the
cation of services. The process is also an effective means by program's success or they should ®nd ways that skills can
which participants and stakeholders can be actively be exchanged.
involved in the process of program planning (Kettner,
Moroney, and Martin, 1990). 4.3. Ecological environment assessment
Opportunities for participation. The main purpose of this
participatory component is to change the perception of Beyond recognizing the individual assets of participants,
program participants as simply individuals with needs to it is also important to consider the impact that the wider
people with assets for the program. In McKnight and Kretz- context, or ecological environment, has on program partici-
mann's (1996) paper they include a survey of community pants. Human service programs that consider the many
members that could easily be adapted by a program to deter- factors that in¯uence the lives of program participants are
mine the assets of program participants. The planning group typically more successful than those that do not recognize
might want to use the inventory not only to determine the these outside in¯uences (Fernandez & Nichols, 1996).
assets of program participants but also of every member of The ecological perspective looks at the relationship
the planning group. between individuals and a variety of system levels. The
To determine assets the planning group could design a levels include: the micro level (such as those family and
survey to include those skills that would be of most use to friends who participants are in direct contact with on a
 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14 7

daily or weekly basis), the meso level (direct contact with ment of participants in the planning process might change
various organizations such as the workplace, school, service after implementation of the program and with the addition
organizations), the exo level (which includes those areas in of new participants. To deal with this problem the planning
which people have indirect contact such as their spouse's group might want to institutionalize a short version of the
workplace or child's school, as well as the neighborhood or ecological environment assessment into new participant
community in which participants live), and the macro level intake or as an on-going component of the program.
(such as the economic and policy context that in¯uences the
participant) (Bronfenbrenner, 1979). 4.4. Program design or replication
It is important that planning groups understand the larger
environment in which program participants live. ªAccord- Examination of already established programs that have
ing to this (ecological) perspective, constructive as well as addressed similar needsÐespecially the program's
dysfunctional behaviors are best understood and most effec- successes and failuresÐcan provide important information
tively promoted, treated or prevented within (or as part of) for planning groups. Replicating other programs may well
the environments where they occurº (Huebner, 1989, be the starting point for a number of groups. Programs are
p. 166). In addition, it is also important to consider any often initiated after hearing about a particularly innovative
factor in the environment which may prevent participants' and/or successful program that administrators believe could
needs from being ful®lled (Mayer, 1985). Knowledge of the be replicated. In some cases, funding sources may require
ecological context of program participants can give clues as that a speci®c type of program be implemented.
to ways that systems outside the program can undermine or However, not all new programs are based on past
support the program. programs. If program planning groups are unsure as to
Opportunities for participation. To understand the ecolo- what type of social intervention may be appropriate to
gical context in which participants live, participants can be address the problem, comparing interventions may prove
interviewed or come together in focus groups to talk about useful (Marsh, 1982; Mayer, 1985). Parts of various
their support systems and the organizations and individuals programs could be combined, or a particular program
they are in contact with in a typical day (for sample could be chosen that may work well based on the need
questions see Appendix B). For example, a program for being addressed, resources available, the ecological envir-
adolescent mothers discovered that the adolescents and onment, and program goals and theory.
their children had frequent contact with the children's Opportunities for participation. When the planning group
fathers who were undermining the parent training goals of is considering the replication of an existing program, parti-
the program. The program was expanded to include fathers. cipants should be given the opportunity to discuss any
It might also be useful for planning group members to adjustments that would need to be made to the program or
visit participants in their home, school, and/or work envir- any potential problems they see with the program as
onments. Stakeholders and participants should be encour- currently operating. This discussion could take place in
aged to consider the ways in which these larger contexts light of what was learned from previous planning compo-
might in¯uence the ability of the program to meet its nents and should consider the uniqueness of the need being
goals by supporting or presenting barriers that could addressed, the purpose of the program, the assets and
in¯uence program success. 3 resources of participants and the community, as well as
Potential pitfalls. There are a number of potential pitfalls the ecological environment assessment. Planning partici-
that planning groups might want to think about before pants might also want to talk with current or past partici-
implementing this planning component. For one, groups pants in the program under consideration to ®nd out what
may have dif®culty ®guring out how to use the information they like best and least about the program. Planners repli-
they learn in the ecological assessment to inform program cating a peer education program used a list of workshop
design. The group should decideÐbefore data collectionÐ topics in other peer education programs to design a survey
what information will be most useful for the planning to measure student interest in those topics (Nichols &
process and how the information collected will be used. Lumley, 1999). They also interviewed other peer educators
This will allow the group to prioritize different parts of about workshop interest and attendance. This information
the planning process. For the component to be useful, the was used to determine program offerings.
planning group must be open to the potential incorporation If the group is attempting to design a unique program,
of different system levels in the program design. If the group participants can be invaluable in providing their visions of a
is not open to this possibility then the assessment could be a `perfect' program. A program that is designed based on the
waste of time and erode trust between the facilitator and the issue to be addressed, the ecological environment, the assets
planning group. of participants, and the resources of the community will
Another potential problem is that the ecological environ- provide the framework for a participatory program. A
program founded on the realities of program participants'
3
For an example of how a program for teenage mothers addressed a lives and planned with and for program participants would
number of systems levels see Fernandez and Nichols (1996). provide an exciting new model of how participatory
8 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14

programs could be planned, implemented, administered, planning steps and allows the group to see how the planning
and potentially institutionalized. Planning groups can start components come together as well as notice any potential
the design of new programs by discussing program theory problems with the theory or the parts of the program.
(the next planning component discussed). Opportunities for participation. Program theory must be
Potential pitfalls. The selection of the program to be understood and resonate with the experiences of program
replicated or designed can be the longest and most dif®cult participants. The planning group must come together at this
planning component to implement. It may also be the most stage and de®ne all the parts of the program, how the
important. Each planning member may have his/her own program uses supports in the ecological context as well as
idea about what program to implement. As a result, tension resources and assets, and what objectives are hoped to result
and the potential for group disintegration is high. The facil- from the combining of these factors.
itator must be con®dent in continually bringing the group Weiss (1997) suggests that when doing theory-based
back to previous planning component work: the issue being evaluation, evaluators should focus on one or two central
addressed, the purpose of the program, the assets and assumptions that underlie the program. Facilitators evaluat-
resources available to the program, as well as the ecological ing a housing program took such an approach by asking
realities of participants' lives. participants the major assumptions they thought drove the
Because the process of deciding on a program design may program and then led the group through a process where
be long, enthusiasm for the planning process may wane at they articulated these ideas into speci®c program parts, and
this point. The group must be encouraged not to replicate a ultimately outcomes. The facilitator can also bring in any
program or choose a particular program just because it is the applicable social science theory (Chen, 1990) and the group
easiest path to take, but struggle with what is the best choice. can use a similar process with that theory as the base of the
Hopefully by this point the group will have become cohe- program design.
sive enough to get past this challenge and ®nd ways to make Potential pitfalls. One of the most challenging parts of
this process enjoyable. The group should know that each implementing this planning component in a participatory
part of the program does not have to be determined at this fashion is explaining to participants and stakeholders what
stage. The next two planning stepsÐthe determination of theory is and its purpose. Facilitators might wish to focus
program theory and program goals/objectivesÐwill help to more on identifying the program assumptions rather than
re®ne the program. telling planning members that the purpose is to determine
a more abstract program theory.
4.5. Program theory Problems may also arise if the group has trouble articu-
lating a particular theory or the assumptions of the program.
Program theory generally helps in the formation and However, once group members begin to see the connections
understanding of a program's goals (Bickman & Peterson, between assumptions and speci®c program components, the
1990). The theory of a particular program is often implicit, group will likely become energized as members witness the
but Chen (1993) argues that it should be more explicit. program taking shape.
Program theory is generally seen as the abstract force and
assumptions that `holds' the program together. Yet because 4.6. Program goals and objectives
theory comes out of experience and can be adapted to create
a better ®t between program practice and the theory under- If done effectively the goals and objectives of the
girding the program, participants need to be involved in program should ¯ow naturally out of the discussion of
de®ning the underlying assumptions of the program being program theory. The de®nition of program goals and objec-
developed. If handled correctly, program theory can help to tives is one area that program planning researchers have
de®ne as well as guide the structure, population, process, universally accepted as necessary to program planning
and outcomes of the program (Conrad & Miller, 1987). (Kettner, Moroney, & Martin, 1990; Mayer, 1985; Rossi
Program theory can be found in a priori theory derived & Williams, 1972; Sylvia, Meier and Gunn, 1991). Not
from academe, from research that helps determine the only does the de®nition of program goals provide a general
underlying causes of a program's workings, or by extraction structure for the program, it also assists in future evaluations
of a stakeholder's implicit theory (Lipsey & Pollard, 1989). of program outcomes. ªGoal-oriented planning forces the
Sometimes the theory comes from the parts of the program. decision maker to think explicitly about desired ends, to
Each program part, when combined with other parts, will confront basic values, and to anticipate the future with
lead to particular outputs and then to immediate, intermedi- longer time perspectiveº (Mayer, 1985, p.113).
ate, and ultimate objectives of the program (see speci®c Yet goals and objectives are often not stated, or not stated
examples of this in Weiss, 1997). At this stage in the plan- in clear terms, which makes the assessment of program
ning process, the speci®c parts of the program can be listed success dif®cult to determine later (Posavac & Carey,
and their connection to one another and the ®nal goals of the 1997; Rossi & Williams, 1972). The program objectives
program can be illustrated. Taking the planning group need to be very speci®c and stated so as to address the
through this process brings together all the previous need(s) or issues that the program was created to address.
 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14 9

Opportunities for program participation. Including For example, participants in public assistance programs are
program participants in the articulation of program goals very aware that the public often de®nes them as `lazy' or
allows participants to understand the importance of each unwilling to work (Edin & Lein, 1997). Stakeholders who
part of the program. To implement this planning compo- are familiar with the community in which the program is to
nent the planning group can review the program as be implemented also often have such insights.
designed in the previous planning step and make any In addition to discussing their perceptions, participants
adjustments to the program that might be necessary for can do some research to look for any local polls or news-
goals and objectives to be achieved. The planning group paper articles that may give some insight into how people
should also come up with goals and objectives that encou- have responded to similar programs or the need that the
rage the continued involvement of current and past program is being designed to address. Facilitators might
program participants as active agents in the administration want to ask a planning member who is the fundraiser for
and evaluation of the program. the program to teach program participants about scouting
Potential pitfalls. As in the case of the other planning for funding. Participants could then get a better sense of
components, planning members may be unable to agree the potential larger support that is available to the
on the goals and objectives of the program. If this happens program and issue. The group can then brainstorm how
there is likely still tension and dissension in the planning to overcome these potential barriers to program imple-
group. If possible, this is the point at which these differences mentation.
in opinion need to be worked out so the program planned is Potential pitfalls. The limits that the political context
coherent and the program design, theory, and goals all ®t places on programs are usually well known to program
together. At this stage the planning group must be cautioned administrators. Often political actors are wary of
that as the program is implemented and administered the supporting a new program before they see evidence of
goals and objectives may change. program success. Further, funding is often limited to
certain types of programs. Too much outreach to poten-
4.7. Policy considerations tial future supporters of the program who need concrete
information about program design, before there is move-
An assessment must also be made of the political climate ment towards implementation may, in some cases, limit
surrounding the need to be addressed by a particular future support from those who may dismiss the program
program as well as the likely political support. In most prematurely. To avoid this possibility the facilitator and
cases offering a new program is a political act, and therefore the planning group might ®rst want to determine the
it is impossible to ignore the political environment (Morill, most prudent way of ®nding out information about the
1989). Understanding the political context and the views of political context without talking to too many potential
the larger society may give some clues as to how accepting supporters. Then, once more speci®cs about the structure
and supportive the public and the community may be of the of the program are known, potential supporters can be
program. contacted. However, each situation is different and the
To assess the policy context planning groups can deter- potential risks and rewards should be understood by all
mine the following: (1) the actors, who will play a role? planning members before moving forward with this
(2) motivations, what do people want? (3) beliefs, what component.
are the value systems of the actors? (4) resources, what
can the actors offer? and (5) sites, what is the context in 4.8. Evaluation plan
which decisions will take place? (Mayer, 1985). The status
of each of these factors as constraints or resources to the Designing a program's evaluation during the program
program need to be contemplated by the planning group. In planning stage is important (Caudle & Newcomer, 1989;
areas where there is little support, especially those areas that Kettner, Moroney, & Martin, 1990). Consideration of the
participants and stakeholders feel are important, extra time evaluation approach early on in the process will provide
and effort may need to be spent to help build political opportunities to design and implement both formative and
support and resources. A discussion of these potential summative evaluations. The planning group can use the
barriers during the planning phase can help to prepare the goals and theory of the program to design monitoring instru-
planning group for any problems that might arise in obtain- ments that can be maintained from the inception of the
ing funding, publicity, facilities, or other resources as well program. The process of developing an evaluation plan
as provide insight into areas of support not previously iden- also helps to educate stakeholders and participants about
ti®ed in the planning process. the importance of program evaluation and provides a
Opportunities for participation. Some understanding of forum around which the planning group can work towards
the political context can be gained through a discussion consensus and the implementation of a participatory
between participants and stakeholders. Program participants program evaluation (see Whitmore, 1998).
have a keen understanding of how others perceive them and Opportunities for participation. Clarifying program goals
any problems they may have encountered in the community. and program theory through an evaluation plan provides
10 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14

another opportunity to make the purposes and desired of many different types of programs. In one sense this is
outcomes of the program explicit. Allowing group members appropriate because each planning group needs to ®nd its
to state their viewpoints and be equally involved in the own way of working together. Participatory program plan-
process of designing the evaluation will help determine ning is by its very nature a process. For each planning
the future roles of planning members in a participatory group it will be an unique process within the context of
evaluation. An evaluation plan that focuses on evaluating the general planning components presented here. Each
program implementation and process as well as outcomes group must ®nd its own way of brainstorming and making
provides planning members with added assurances that decisions that is most appropriate given the members of the
program adjustments can be made if the program as initially planning team, the program being planned, and the plan-
planned is not accomplishing desired results or meeting the ning context.
goals of the program. The components included in this planning model provide
One of the greatest bene®ts of including participants in a foundation from which a program can be planned. They
the evaluation plan is that the probability that evaluation also provide opportunities for participants and other stake-
protocols will be followed and utilized is increased (Greene, holders to be actively involved in the planning process. And
1987). Participants will learn early on how to utilize although the role of evaluators becomes complicated when
program evaluations as tools for program improvement. A they are involved in program planning, there is no doubt that
neighborhood revitalization program designed a participa- evaluators know from experience what is necessary for
tory evaluation that both tracked and monitored program programs to function effectively. Utilizing this knowledge
outcomes and impact as well as provided training in evalua- will be very valuable for the development of future
tion techniques that ultimately empowered participants programs.
(Folkman & Rai, 1997). Involving participants in the As this planning model is implemented in various
evaluation design during program planning will help in contexts the details and intricacies of each program plan-
the collection of baseline and consistent data as well as ning component will become more clear. With each
encourage that program goals and objectives are being program planned, similarities in processes, problems, and
assessed. outcomes will likely become apparent providing further
Potential pitfalls. As experienced by many programs, guidance for future program planning groups. Hopefully
there may be a lack of resources to conduct evaluations this article will help to encourage dialogue around these
(particularly outcome evaluations). If there is no support issues and inspire others to share the lessons they have
up-front the group might want to rely on the program plan- learned from participatory program planning.
ning facilitator to conduct the evaluation. However, this is
not idealÐthe facilitator will need to emphasize all that can
be learned from someone who has not been involved in the
6. Conclusions
program planning process. The group should be encouraged
to seek out funding for this purpose.
The program planning model presented in this paper is a
There also could be resistance by the planning group to
comprehensive planning process with many opportunities to
implementing changes to the program that might be indi-
involve participants in program design. Including future
cated from results of the formative evaluation. Given the
program participants along with other stakeholders in the
time and hard work that the group put into planning the
planning process can help in the development of a program
program, the members might be hesitant about changing
that participants will use, that stakeholders and participants
the program in any way. The importance of making changes
feel invested in, and that can successfully be evaluated after
that will better enable the program to meet its goals can be
the program is implemented.
the force that convinces the group of the need to make
Combining the knowledge of program administrators and
adjustments.
program participants is certainly a challenge. However,
evaluators are uniquely suited to orchestrate such a process.
And including the needs and wants of potential program
5. Lessons learning participants in any approach undertaken to change their
situation may well be worth the effort.
Because of the newness of this program planning model,
there is still much to be learned and explored about program
planning in a participatory manner. The planning model
combines two major areas that would each bene®t from Acknowledgements
further exploration: program planning and participatory
planning. Therefore, more research and case studies of parti- Thanks to Subhash Sonnad, Huey Chen, Marilyn Fernan-
cipatory program planning are necessary. dez, Sandra Bever, Deborah Lustig, and incredibly helpful
The planning model presented in this paper has been and insightful anonymous reviewers for their contributions
designed to be general enough to be used in the planning to this paper's development.
 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14 11

Appendix A

Summary of program planning components and examples

of their application in two context is given in Table A1.
Table A1

Component Purpose of component Peer education program Transitional housing program

Participatory input To include those most affected by a Participants: Female students living on campus Participants: Single parents living in a
program in the planning homeless shelter
Stakeholder check-In To include key stakeholders with Administrators, resident directors, women's Program administrators and staff,
power and resources needed to resource staff shelter director, homelessness task
implement the program force
De®nition of need To speci®cally de®ne the need or Focus groups with female students who are Participants say that single parents
and program purpose challenge to be addressed by the Resident Assistants reveals that women have a dif®cult time moving from
program and its ultimate purpose students experience low self-esteem and high homeless shelters to permanent,
rates of eating disorders and acquaintance rape stable housing
Survey of campus clubs shows that few female Data show that single parents who are
students hold leadership positions on campus low-income often have low levels of
job experience and education
Constant moving disrupts homeless
children's ability to do well in school
Resource and asset To determine what resources and Women students who are interested in working Residents of the program are
mapping assets are available for utilization on the program are interviewed to determine interviewed at intake to determine the
by the program from participants the skills they have to contribute to the skills and experience they have to
and the community program offer the program
Students map the resources and assets Residents work with case workers and
available to women students on campus community members to map the
programs and services available to
them in the surrounding community
Ecological To determine the purpose of the A random sample of women students are At intake new residents are
environment program within the context of interviewed about where they get support interviewed where they get support
assessment program participants lives (to Students who have successfully overcome Residents brainstorm about ways the
include but not limited to: family eating disorders and senior students with high program can help them access
and friends, work, neighborhood, involvement are interviewed about what is resources at various systems levels
other community organizations) helpful to them (including other parent in trainings,
The group considers how to include friends, providing mentors)
faculty, and parents into program design Barriers outside the program that
might prevent participants from
meeting program goals are discussed
and identi®ed by participants
Program design or To determine what type of program Students brainstorm types of programs that Residents brainstorm a program
replication might be most appropriate to plan would address needs structure that will address the
given the situation to be addressed, After agreeing on replicating a peer education challenges and issues they face
the needs, assets, and ecological program (introduced and encouraged by Four residents are chosen to be
environment college administrators), students research peer representatives of the group. The reps.
education programs at other colleges meet with the evaluator and key
Focus groups are held with RA's to get their program staff to brainstorm potential
opinions of instituting a peer education program structure
program and to help design the survey for Group decides on a program that
students combines quality, two-year housing
A random sample of residential female with program supports for job
students are surveyed as to program topics they training, child care, support groups,
would be interested in learning more about and resident-centered case
management
Program plan is taken back to all
residents, adjustments to program are
made, assignments to particular
leadership roles in each program
component are assigned
12 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14

Table A1 (continued)

Component Purpose of component Peer education program Transitional housing program

Program theory To de®ne the explicit components Participants discuss and agree on program Participants discuss and agree on
of the program and the assumptions assumptions/theory: assumptions of the program:
underlying its success Students learn best from fellow students Two years is enough time for
(Sagaria & Johnsrud, 1988) residents to learn skills to make them
Students will attend workshop topics that better able to transition from
interest them homelessness to permanent housing
Peer educators will learn leadership skills Children in stable housing will do
better in school
Program goals To create agreed upon program Participants and key stakeholders decide to Representative participants and
goals and objectives that clearly Provide at least 15 residential peer-led stakeholders agree on goals:
de®ne the purpose of the program workshops a year on the major topics listed in Residents will be able to live in
the survey as of interest to students housing for up to two years as long as
Train at least ten students a year as peer they do not violate agreed upon rules
educators of the program
Improve the knowledge of students who attend Program staff agree to include
the workshops by at least 20 percent participants in program decision
Workshop leaders will meet once a year to making
review evaluation outcomes and re®ne the All residents will have a case plan that
program they will follow
Residents will have more marketable
job skills when they leave the
program than when they began
Children in the program will increase
their grades and social skills
Eighty percent of former program
residents will be able to successfully
transition to permanent housing
Policy considerations To identify the larger macro Talking with administrators, participants ®nd The planning group discusses recent
structures such as funding and out that the university likes the program changes as a result of welfare reform
outside support that may determine because of its low cost that will affect participants' ability to
the ability of the program to be In a planning session it becomes evident that participate in the program. They
created and sustained. To make support from residence hall staff and student research public polls that show
contact with those individuals and leaders is key to the implementation of the support for programs that promote
groups whose support is needed for program. Students brainstorm ways to include `self-suf®ciency'
the program to be implemented more residential leaders as stakeholders and Through discussions with program
participants. Resident Directors are added to administrators and research of
the planning group funding options for the program,
participants ®nd that housing upkeep
and maintenance is expensive and
that grants to cover such expenses are
dif®cult to ®nd. Brainstorm ways to
overcome these limitations
Evaluation plan To develop a plan for an on-going Students assist in developing the evaluation Participants assist in developing the
participatory evaluation of the plan. Pre and post tests measuring topic evaluation plan
program as well as a time line for knowledge and leadership development are What will be tracked through case
external, outcome evaluations created at the same time as the workshop ®les is decided upon (contract,
curriculum training/educ., children's grades)
A method of keeping track of number of A method for staff to ®ll out detailed
programs offered, attendance, and satisfaction reports whenever anyone is asked to
with the program is developed leave the program is developed
Time line for external evaluation is
decided upon
 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14 13

Appendix B References

Sample interview guide for ecological assessment is Ayers, T. D. (1987). Stakeholders as partners in evaluation: A stakeholder-
given in Table B1. collaborative approach. Evaluation and Program Planning, 10 (3),
263±271.
Table B1 Baca Zinn, M. (1989). Family, race, and poverty in the eighties. Signs, 14,
856±874.
System level Possible interview questions and their Bickman, L., & Peterson, K. A. (1990). Using program theory to describe
p
Connection to the program and measure program quality. New Directions for Program Evaluation,
47, 61±72.
Microsystem Name family and friends who you are in contact Bronfenbrenner, U. (1979). The ecology of human development, Massachu-
with: Every day, once a week, once a month. What setts: Harvard University Press.
types of support do they provide you? In what Caudle, S. L., & Newcomer, K. E. (1989). Political ends and practical
ways might they hinder your progress? Will they needs: A case study of a pilot program. Evaluation and Program Plan-
support your participation in the program? ning, 12, 279±286.
p
The planning group should use information from Chen, H. T. (1990). Issues in constructing program theory. New Directions
these questions to determine if the program needs for Program Evaluation, 47, 7±18.
to involve other members of the microsystem Chen, H. T. (1993). Emerging perspectives in program evaluation. Journal
important to the participant in the program design
of Social Service Research, 17 (1/2), 1±17.
Mesosystem What kinds of outside organizations are you in Collins, P. H. (1990). Black feminist thought: Knowledge, consciousness,
direct contact? (Probe for school, work, health and the politics of empowerment., Boston, MA: Unwin Hyman.
care, religious, public transportation, libraries, Conrad, K. J., & Miller, T. Q. (1987). Measuring and testing program
other social service agencies, etc.) How often are philosophy. New directions for program evaluation, 33.
you in contact with these organizations? What Cook, S. W. (1988). The 1954 social science statement and school deseg-
type of support do they provide you? regation. In D. A. Taylor, Eliminating Racism (pp. 237±256). New
p
The planning group should determine if these York: Plenum Press.
organizations can help support the participant in Doherty, J. E., & Rispel, L. C. (1995). From con¯ict to cohesion: Involving
program activities or if they hinder the stakeholders in policy research. Evaluation and Program Planning, 18
participantÐfor example if transportation is not (4), 409±415.
accessible the program might need to address Edin, K., & Lein, L. (1997). Making ends meet: How single mothers survive
these limitations. (That is provide transportation welfare and low-wage work, New York: Russell Sage Foundation.
for evening activities, etc.) Fernandez, M., & Nichols, L. (1996). Ecological approach in practice: A
Exosystem Describe the neighborhood where you live. Where case study of the ounce of prevention fund. Journal of Sociology and
do people in your neighborhood work? Do they Social Welfare, 23, 121±141.
look out for each other, know each other, etc.? Are Fishman, D. B. (1992). Postmodernism comes to program evaluation: A
they single parents? Is there much crime? Do you critical review of Guba and Lincoln's fourth generation evaluation.
feel safe in your neighborhood? Do you have Evaluation and Program Planning, 13, 263±270.
access to resources you might need to supplement Folkman, D. V., & Rai, K. (1997). Re¯ections on facilitating a participatory
program activities? What organizations that your community self-evaluation. Evaluation and Program Planning, 20 (4),
spouse/parents/children are involved with might 455±465.
limit or support you in participating in the Freire, P. (1970). Pedagogy of the oppressed, New York: Continuum.
program? Gergen, K. J., & Gergen, M. M. (1991). From theory to re¯exivity in
p
The planning group needs to determine if the research practice. In F. Steier, Research and re¯exivity, Newbury
larger context in which participants live will Park: Sage Publications.
undermine the goals of the program Greene, J. C. (1987). Stakeholder participation in evaluation design: Is it
Macrosystem How do welfare, food stamp, health care, taxes, worth the effort? Evaluation and Program Planning, 10, 379±394.
crime, etc. policies impact on your life? (Policies Greene, J. C. (2000). Challenges in practicing deliberative democratic
that in¯uence participants in the program and their evaluation. New Directions for Evaluation, 85, 13±26.
ability to participate in the program should be Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation,
explored here.) Is social class or race a factor? Newbury Park: Sage Publications.
p
The planning group should determine if there are Hallett, M. A., & Rogers, R. (1994). The push for truth in sentencing:
requirements of policies that affect participants Evaluating competing stakeholders constructions. Evaluation and
that will help or limit the program in meeting its Program Planning, 17 (2), 187±196.
goals. For example, for a program with Harris, S. M. (1995). Cultural concerns in the assessment of nonwhite
participants who receive welfare bene®ts, the students' needs. In S. Stabb, S. Harris & J. Talley, Multicultural
program might require that participants go to needs assessment for college and university student populations (pp.
school full-time but under welfare reform, 17±50). Spring®eld: Charles C Thomas Publisher.
participants have mandatory work requirements Hodgson, G. (1978). America in our time, New York: Vintage.
that may limit their ability to attend school or they Huebner, L. A. (1989). Interaction of student and campus. In U. Delworth &
may be eligible for scholarships G. R. Hanson, Student services: A handbook for the profession (pp.
165±208). Jossey-Bass Publishers: San Francisco.
Kelly, K., & Van Vlaenderen, H. (1995). Evaluating participation processes
in community development. Evaluation and Program Planning, 18,
371±383.
Kettner, P. M., & Daley, J. M. (1988). Designing effective programs. Child
Welfare, LXVII (2), 99±111.
14 L. Nichols / Evaluation and Program Planning 25 (2002) 1±14

Kettner, P. M., Moroney, R. M., & Martin, L. L. (1990). Designing and Papineau, D., & Kiely, M. C. (1996). Participatory evaluation in a commu-
managing programs: An effectiveness-based approach, Newbury Park: nity organization: Fostering stakeholder empowerment and utilization.
Sage Publications. Evaluation and Program Planning, 19 (1), 79±93.
Lipsey, M. W., & Pollard, J. A. (1989). Driving toward theory in program Posavac, E. J., & Carey, R. G. (1997). Program evaluation: Methods and
evaluation: More models to choose from. Evaluation and Program case studies, Upper Saddle River, NJ: Prentice-Hall.
Planning, 12, 317±328. Reason, P. (1988). Human inquiry in action, Newbury Park: Sage Publica-
Marsh, J. C. (1982). Public issues and private problems: Women and drug tions.
use. Journal of Social Issues, 38 (2), 153±165. Risman, B. J. (1993). Methodological implications of feminist scholarship.
Martinson, R. (1974). What works? Questions and answers about prison The American Sociologist, Fall, 15±25.
reform. The Public Interest, 35, 22±54. Rixecker, S. S. (1994). Expanding the discursive context of policy design:
Mathie, A., & Greene, J. C. (1997). Stakeholder participation in evaluation: A matter of feminist standpoint epistemology. Policy Sciences, 27,
How important is diversity?. Evaluation and Program Planning, 20 (3), 119±142.
279±285. Rossi, P. H., & Williams, W. (1972). Evaluating social programs: Theory,
Mathison, S. (1994). Rethinking the evaluator role: Partnerships between practice and politics, New York: Seminar Press.
organizations and evaluators. Evaluation and Program Planning, 17, Ryan, K. E., & Johnson, T. D. (2000). Democratizing evaluation: Meanings
299±304. and methods from practice. New Directions for Evaluation, 85, 39±50.
Mayer, R. R. (1985). Policy and program planning: A developmental Sagaria, M. A. D., & Johnsrud, L. K. (1988). Generative leadership. In M.
perspective, Englewood Cliffs, NJ: Prentice-Hall. A. D. Sagaria, Empowering women: Leadership development strategies
McKnight, J. L., Kretzmann, J. P., (1996). Mapping community capacity. on campus (pp. 13±26). San Francisco, CA: New Directions for Student
Northwestern University, IL: Center for Urban Affairs & Policy Services.
Research. http://www.nwu.edu/IPR/publications/papers/mcc.pdf. Swantz, M. L., & Vainio-Mattila, A. (1988). Participatory inquiry as an
Morgan, G. (1983). Research strategies: Modes of engagement. In G. instrument of grass-roots development. In P. Reason, Human inquiry in
Morgan, Beyond method: Strategies for social research (pp. 11±42). action (pp. 127±143). Beverly Hills: Sage Publications.
Beverly Hills: Sage Publications. Sylvia, R. D., Meier, K. J., & Gunn, E. M. (1991). Program planning and
Morill, W. H. (1989). Program development. In U. Delworth & G. Hanson, evaluation for the public manager, Prospect Heights: Waveland Press.
Student services: A handbook for the profession (pp. 420±440). San Weiss, C. H. (1972). Evaluation research: Methods for assessing program
Francisco, CA: Jossey-Bass. effectiveness, Englewood Cliffs, NJ: Prentice-Hall.
Nichols, L., & Lumley, L. (1999). Involving students in the development of
Weiss, C. H. (1983). The stakeholder approach to evaluation: Origins and
a peer education program for college women. Journal of College
promise. In A. S. Bryk, Stakeholder-based evaluation (pp. 3±14). San
Student Development, 40, 422±427.
Francisco, CA: Jossey-Bass Inc.
O'Sullivan, R. G., & O'Sullivan, J. M. (1998). Evaluation voices: promot-
Weiss, C. H. (1997). Theory-based evaluation: Past, present, and future.
ing evaluation from within programs through collaboration. Evaluation
New Directions for Evaluation, 76, 41±55.
and Program Planning, 21, 21±29.
Whitmore, E. (Ed.). (1998). Participatory evaluations. New Directions for
Palumbo, D. J., & Hallett, M. A. (1993). Con¯ict versus consensus models
Evaluation, 80.
in policy evaluation and implementation. Evaluation and Program
Planning, 16, 11±23.