TRAINING MANUAL

FOR

AN ECOSYSTEM MODEL

Stage II

What to Assess

The determination of which environmental interactions or characteristics to assess will be the planning team's goal in Stage II. As the team discusses what it wants to learn from the ecosystem assessment, the conversation inevitably will turn to the school's values and goals and how these have been transmitted through policies and programs active in the environment under study. Therefore, the team enters the ecosystem model on step five, measuring student perceptions of the environment, but in so doing the team will also deal in some depth with the model's first four steps concerning the values, goals, and resulting environmental conditions that reflect these.

The decisions on what to assess should not be rushed because they will lay the foundation for the remainder of the team's work and will directly influence the type of data the model will procure for subsequent environmental redesign. Still, there is great temptation on the part of teams to overload the processes. Care should be taken to keep the discussion moving in a productive manner.

The team's first task will be generating ideas on what to assess. The next task that usually arises is the need to validate these ideas and make certain that the team has not overlooked an important environmental transaction or characteristic which should be assessed. The final task in Stage II is choosing the assessment categories which are needed and setting boundaries for the assessment.

The processes presented in Stage II are suggested as aids to help the team generate, organize, and moderate its flow of ideas on what to assess. The processes are given in the order in which most teams have applied them. However, it is common for a team to cycle through the processes several times, often trying different approaches, as it comes to a final determination on what should be assessed. Thus the processes interact with one another as the team clarifies the priorities, limits, and restraints for an ecosystem assessment.

Generating Ideas

Discussion

Some ideas about what the project should assess will have been discussed during the team's initial meetings. These ideas usually have been articulated in general terms and cover only the more obvious areas of inquiry. Thus the team's first task is to start generating more ideas and then to strive for greater specifics. It is not unusual for a team to experience some difficulty in its attempts to become more specific. But once a team starts asking questions it would like answered about particular environmental conditions or transactions at work within a general area of inquiry, the team will have overcome this problem. The team will then have to strike a reasoned balance between the need for obtaining specific assessment questions and the temptation to make the list of questions too long. A useful system of checks and balances is to examine periodically the team's ideas and select those that are the most important.

A problem that may be encountered early in the process is the voicing of concerns over what can be assessed. There will be areas of inquiry that one or another of the team members will reject as a subject for assessment because the area is considered to involve policies or conditions that cannot be changed. When this occurs, it is advisable for the team to note the rejection and move on to another area of inquiry. In this way the issues can remain flexible. Then, as team members consider other areas of inquiry and discuss how conditions in these areas could be improved if the proper information were known, the advantages of an ecosystem assessment become more exciting and better understood. This dynamic often results in a team member reopening discussion on an area of inquiry he/she had originally rejected. Ensuing discussion can then set realistic limits on what can and cannot be assessed within the area of inquiry.

An important topic often raised during a team's discussions on what to assess is how the assessment will be accomplished. The third stage of this manual concentrates on this topic so that the ultimate selection of assessment techniques can be more closely matched to the team's defined assessment needs. When the subject of assessment techniques is raised during Stage II, it should remain subordinate to the main topic--what to assess--and be guided by the fact that this model will emphasize the use of assessment instruments on which respondents write their answers. While individual and small-group interview techniques and behavioral observation are valid and productive methods, they are impractical if many people within the environment's population are to be assessed. Therefore, it is suggested that teams who want to use such techniques do so as additional measures to written instruments.

Process

The brainstorming process outlined in Technical Appendix B, p. 85, can be usefully employed to generate ideas for assessment. It may take several brainstorming sessions to free-up thinking and to surface the variety of ideas needed to tap information on all the transactions that take place in the environment under study. Each session should be reviewed. The team should strive for a degree of specificity. A normal progression from an initial suggestion, Why do students break residence hall contracts? might be suggestions such as, Would students use sublet clauses? Are students willing to pay for additional services? and How do students learn about contract policy? during subsequent brainstorming sessions.

Once ideas run dry or become so specific as to be impractical, the process should be halted. Other indications that the process should cease are repetition of ideas or persistent rephrasing of the same idea. The team's effort are better directed toward other facets of the task such as grouping or categorizing the ideas and selecting those ideas that seem most important. Therefore, the review of each idea-producing session should be conducted with an eye on achieving specificity without letting the list of ideas grow so long as to become unmanageable.

The brainstorming process can be begun in a number of ways. Two suggested methods are:

1. The team brainstorms replies to the question, What do we want the data to tell us when we are finished? Example: Residence hall environment. Sample replies from first brainstorming session:

Why do students terminate living in the residence halls?

Do students like the maintenance service?

Which programs do students like?

Why do students change residence halls?

Would students be willing to pay more for better accommodations?

After the first brainstorming session, the team should try grouping their ideas. As various groupings are made, suggestions for category headings will emerge. The grouping of ideas into categories will help the team identify additional categories or areas of inquiry and become more specific as it conducts subsequent brainstorming sessions for each category.

Example: the sample of above replies grouped and categorized:

Decision Making

Why do students terminate living in the residence halls?

Why do students change residence halls?

Facilities

Would students be willing to pay more for better accommodations?

Do students like the maintenance service?

Programs

Which programs do students like?

2. The team makes a list of environmental categories or areas of inquiry that its members would like assessed and conducts a brainstorming session for each. Example: Community college counseling center system:

Category One--What do we want to know about our students? Sample replies from first session:

Where do students go for information?

What types of information do they need?

Category Two--What do we want to know about student services?

Category Three--What do we want to know about our teaching environment?

After a brainstorming session has been completed for each of the environmental categories or areas of inquiry, the team should review all the replies. Since some ideas may not always fit the category for which they were given, the need to reassign ideas to another category may occur.

Subsequent brainstorming sessions can be done for categories short of ideas or for all the categories. It may be helpful to arrange the replies generated in each session in columns side by side. This not only gives the team a ready review of their work, but it also provides a visual illustration of how subsequent sessions produce more replies with greater specificity. Example: Sample replies from two brainstorming sessions on

Category One given above.

First Session Replies

1. Where do students go for information?
2. What types of information do they need?

Second Session Replies

1. Are advisors available when students need them?
2. What is frustrating for students?
3. Is the college catalogue helpful?
4. How do students get information about choosing classes in their major?

Whatever method the team uses to initiate ideas on what to assess, periodically it should review the ideas in an attempt to determine if another approach could be more productive. Example: Community college counseling center system. The ideas generated during the first several brainstorming sessions reflected four areas of inquiry important to the counseling center system. These were: students--who they are and what they are like; goals--why students come to the college; college--its curriculum, services, policies, and peer groups; and the interactions of students, goals, and college. Thus subsequent brainstorming sessions focused on the new approach and its four categories.

* * *

Validating Ideas

Discussion

Once the team has generated a number of ideas and begins to formulate a working set of assessment categories, the need may arise to validate its work in one way or another. A suspected constraint on the proposed assessment, such as a policy, program, or physical property within the environment that should not be a subject of inquiry because no subsequent action could be taken to modify or change it, might need to be investigated. A team member may want to become more familiar with the environment in order to know what to assess or a team member may wish to become more familiar with specific transactions in the environment in order to identify whether or not vital assessment points have been overlooked; or, team members may want to determine whether they have covered all the areas of inquiry that should be covered.

Process

When this occurs, the opportunity for individual assignments and some communication with the project's constituents arises. Among some of the methods that might be used are to:

1. Assign the team member in the best position to obtain information about suspected constraints to explore what flexibilities might exist. For example, could the residence hall system offer better accommodations, and what kind, if students were willing to pay more? The team member assigned to explore this issue determines that the residence hall system could not accommodate individual air conditioners, hot plates, or refrigerators. It could accommodate more frequent linen and maid service and phones in individuals' rooms.

2. Assign those team members unfamiliar with the environment or a key transaction within it to explore the environment or experience the transaction personally. For example, have the member(s) attend a residence hall meeting, walk through several halls at lunch time or in the evening, observe what is going on, and talk with residents. Or have the member(s) actually experience transactions in the environment by assuming a key role. For example, the member assumes the role of a student, obtains necessary materials, and goes through registration or processes a course drop/add.

3. Develop a very short questionnaire that each member could ask several of his/her constituents to answer regarding the team's areas of inquiry in order to validate the team's categories and to determine if any areas have been overlooked. Community college counseling center system constituent questionnaire example:

What do you think are the school's greatest strengths?
What do you think are the school's greatest weaknesses?
What do you personally find frustrating around here?
What do you personally find satisfying around here?
Have you had any contact with the Counseling Center?
What do you like most about the Center and its counselors?
What do you like least about the Center and its counselors?
What other issues about school are important to you?

Responses are compared with the team's areas of inquiry and additions or deletions made as appropriate.

* * *

Choosing Assessment Categories and Setting Assessment Boundaries

Discussion

Usually, discussions regarding constraints and the regrouping and categorizing of ideas will provide a preliminary narrowing of choices that the team needs to make regarding the scope of its assessment. Questions will have been raised and some decisions made about how many areas should be assessed and to what degree. It is possible that these discussions will have moderated the team's choices to the point that what it wants to assess is feasible to assess. However, inexperience with the model coupled with enthusiasm for the opportunities it opens to improve environmental conditions more often leads the team to exceed the model's optimum capability.

The single greatest constraint on the ecosystem model at this stage of its development is that it is time consuming to conduct and respond to its assessment techniques. If the assessment is too long, the respondents tire and the quality of information dramatically declines. Its greatest virtue should be that every question has utmost importance to the environment and thus respondents are not offended by the questions. A real hope for the model's future is that, as assessment of an environment is repeated on a campus, a core of environmental referents will emerge that can be subsequently incorporated into a mechanized test instrument. In this case, only new environmental referents will have to be sought by separate procedures.

Because the first application of the model does rely on rather involved procedures to obtain environmental referents, assessment categories and items must be held to a realistic number. In order to develop practical assessment instruments, the team will have to devote time and energy to deciding what is most important and beneficial to learn from the assessment. It will have to set priorities and, ultimately, boundaries for the project.

The planning teams that have used this ecosystem model usually begin the task by reviewing the assessment categories that they have identified to determine those the team feels are the most important. Next, they have reviewed the ideas in each category to determine which of these need to be included in order to obtain the necessary information about the category. For some categories this can be accomplished with only a few questions; for other categories, it may take a number of questions to cover adequately the area of inquiry and obtain the necessary information. As the team reviews the assessment categories and items and sets its priorities, the assessment boundaries will be selected and established.

The team enters this process with at least one important boundary or parameter established, namely, who the assessment's respondents are to be. Other important boundaries to be established are the number of categories to be investigated,(the assessment's scope), the number of questions to be asked under each category (the assessment's level of inquiry), and the constraints which must be honored from both environmental and model viewpoints. Common among these constraints will be computer capabilities and resources, monetary and manpower resources, and subjects of inquiry that involve values, goals, or physical properties which cannot be changed.

Setting priorities for areas of inquiry will often necessitate trade-offs among members of the team. Consensus rapidly develops over the importance of several categories. Then the particular interests that each member represents will emerge. The team will have to find ways to satisfy these claims while maintaining its goal to keep the assessment within reasonable limits. Accommodation and creativity is paramount.

This is also a point at which a team member or consultant knowledgeable in computer technology and assessment techniques can provide the team with real help. Such a person can give the team guidance on computer capabilities and constraints as well as judgment on the assessment's ultimate length if this or that is included or dropped. And there are a variety of ways to obtain information. The team will need to know what alternatives could be used. It may well be that items can be incorporated into the assessment that will secure needed information without having to develop the item's full category. In this way some issues important to a team member can be retained even though the category may be dropped.

Process

The team's first order of business in choosing assessment categories is to assign each category or area of inquiry a rating of relative importance

1. Each team member might be asked to rank order the categories. The categories can then be ranked according to team consensus.

2. Or the team may prefer to use a field force analysis (Technical Appendix B, p. 87) on each category to determine the rank order it should have.

Once each category's priority has been set, the team will need to decide how it can streamline or reduce the number of questions it wants to ask within each category.

1. One category which easily can be streamlined is demographic data about respondents. Since it is common for research to deal extensively with demographic data, team members often have an arm's length of items reflecting these interests. However, in the ecosystem model only a few demographic statistics will be needed for the purposes of separating responses by key groups (such as academic majors or night and day students) in order to compare these responses with the total group surveyed, or for the purposes of comparing the responses of two subgroups (such as women and men). Other demographic differentiations might be place of residence, age group, ethnic background, or year in school, depending upon the environment under study.

a. Look at demographic items in view of environmental design and ask if it really is essential to know the information to plan changes. Remember that most changes will stem from the information gathered through environmental referents--the respondents ' written replies about why they have their perceptions.

b. Look for alternate ways to obtain desired demographic data. For example, residence hall environment demographic data could be pared to sex, residence hall, and academic major. In some cases, the place of residence might also be used to delineate freshmen, graduate students, or married students.

2. Another useful streamlining method is to rank each category's ideas or items and drop the least essential ones or conduct a field force or similar analysis on each to determine which would be most useful to retain .

3. Finally, the team may need to drop categories. If agreement cannot be reached, the team should look for acceptable accommodations.

a. Retain what appears to be the category's most important item in order for the assessment to establish whether the category is critical. If response indicates the item is critical, then a follow-up study could be conducted.

b. Determine if the items of a category are reflected by items in other categories. If so, choose the category most useful for design purposes. For instance, a list of information needs pertinent to residence staff may also be reflected in items suggested under staff/resident interactions. The interaction category ultimately will hold more value for design purposes. The team is best advised to drop the first category--residence staff in favor of the staff/resident interaction category.

c. Finally, if the team members or consultants knowledgeable in assessment techniques advise that the team still has more information desires than can be answered by respondents in a reasonable length of time--usually an hour to an hour and a half--the team should take action and drop as many low-ranking categories as needed to bring the assessment within time limits. An alternate course of action might be to let these categories and their items remain, proceed with Stage III processes for developing the assessment instrument and eliminate based on results of the instrument pilot test. A consideration which should be given to this approach is the fact that it can add significantly to the team's work in Stage III.

Stage II processes provide the team with a set of guidelines necessary for developing an assessment instrument. Once the team has established what it will assess specifically, the scope and level of inquiry, and who will answer the assessment, it is ready to begin Stage III.