TRAINING MANUAL

FOR

AN ECOSYSTEM MODEL

Stage IV

Assessment and Analysis

The implementation of an ecosystem assessment and analysis involves the participation of many people and the coordination of a number of tasks. The largest group of participants will of course be the assessment's respondents. Those who conduct the computer scoring and analysis of data from the phase one instrument will be important participants. The team may well need assistance in implementing the assessment. It may also want these implementers to do some portion of the ER analysis. Therefore, implementers are another set of important participants. Staff or personnel working within the environment will be indirect participants, if not direct participants in some capacity.

The team's major overriding task in Stage IV, then, will be coordinating and overseeing the work and activities of the project's many participants. An adjunct task will be informing everyone in the environment--including staff--about the project and why the assessment is being taken. An obvious task will be selecting and contacting the assessment's respondents. If implementers are used, most likely they will need some training, and arrangements will have to be made for the assessment's testing sessions. The team will also need to keep a close liaison with its computer personnel and its team members' constituencies.

In Stage IV these tasks are reviewed from the standpoint of setting up an assessment schedule and then discussed in more detail with suggested processes that the team can use to coordinate activities, implement the assessment, and conduct data analysis. The team will need to decide which of its members will take responsibility for which of the tasks it schedules. An adequate period of time should be assigned for the completion of each task. Experience with the model indicates three to four months are needed to successfully implement the assessment and conduct the first analyses on its data.

Setting an Assessment Schedule

Discussion

The team will find that the implementation of its assessment technique and analysis of data will go more smoothly if it sets up a schedule and assigns members to oversee the completion of each task. A governing factor in setting up a schedule will be choosing the most opportune time to give the assessment instruments. Respondents need to be familiar with the environment. The influx of new people each semester or quarter, especially in the fall, necessitates a settling-in period before everyone has experienced an environment long enough to be able to give an informed judgment concerning its impact with adequate environmental referents.

This consideration suggests that eight weeks into a quarter or semester is a good time to begin the assessment. The team will also need to consider whether the design or change of the environment that will result from the assessment can be made in time to evaluate its impact on the same population. The ecosystem model is best used when its full cycle is applied to the same population in an environment. Therefore, an ideal schedule for the academic year would be a mid-autumn administration of the assessment technique, with initial analysis of data completed by early winter so that any subsequent redesigns could be in place by early spring and evaluated near the close of the academic year.

This ideal schedule will need to sequence properly with another obvious consideration in choosing the opportune time to conduct the assessment. Respondent attendance at testing sessions can be seriously eroded if these sessions are scheduled too near examination periods. Computer time may be at a premium during these periods, too, if the team's computer facilities are also used for scoring school exams and recording student grades. Thus any schedule must take these factors into account as well.

After the team has determined the best time to conduct its assessment, it can then work backward to schedule the tasks it needs to complete before giving the assessment instrument. In a similar manner, the team can also schedule data analysis tasks in accord with the date that the computer scoring of data will be available. Depending upon the computer situation, the team may want to proceed with the analysis of ER information while awaiting the computer analysis of phase one instrument data.

Process

In setting up its data collection and analysis schedule, the team will have a set of tasks generic to the ecosystem model, as well as tasks unique to its situation and project. Tasks common to the model that need to be scheduled are listed below in their recommended sequence. To establish the time line for each of these tasks, it is suggested that the team briefly review the material given on each of these tasks in the remaining sections under Stage IV. The team should incorporate into its schedule the other tasks unique to its situation as appropriate.

1. Selection of respondent sample
2. Arrangements for testing procedures
3. Communication with key personnel
4. Recruitment and training of implementers
5. Project publicity
6. Respondent contact and scheduling
7. Assessment testing sessions
8. Computer scoring of data
9. Scoring and analyses of ER data
10. Initial comparative analysis on ER and computer data
11. Initial report on assessment

Selection of Respondent Sample

Discussion

The team must determine both the size and composition of its respondent sample. The sample must be large enough to be representative of the entire population and should include an adequate number of respondents from key demographic categories on which the team will want to compare data. Taking a random sample of the entire population or a random sample within demographic categories (e.g., a sample from the residence hall system versus a sample from each residence hall) will accomplish this.

A stratified random sample can be utilized when the team wants to assure that the percentage of respondents in the sample is equal to the percentage of persons in that demographic category within the total population. Thus, if 25 percent of the entire population are women, the team would choose a random sample of women to equal 25 percent of the group to be assessed.

Generally, at least a ten percent sample of the population is desired. A higher percent is always desirable and should be possible when the population is small. It is usually important to obtain a sample in excess of ten percent if a random sample within demographic categories is the chosen method (e.g., each residence hall).

Process

To select a respondent sample the team will need to:

1. Determine what percent of the entire population is most feasible to test. Generally at least a ten percent sample of the population is desired. A higher percent should be possible when the environment's population is relatively small.

2. Determine the ratio of respondents that will be needed from the total sample for each demographic breakdown on which data will be analyzed. Colleges often routinely collect such data (e.g., percentage of day versus evening students).

3. Once the total number of respondents has been set with the corresponding ratios of people needed to represent demographic breakdowns, a random or stratified random sample should be selected from among the possible respondents. Usually, this is a simple procedure, for most schools have data banks from which a random selection of respondent names and addresses can be pulled that accommodate the team's desired needs. In some situations, notably colleges composed of commuter students, the only feasible testing situation will be during class sessions. In this case, it is important that classes either be chosen randomly or in a careful selection process so that class members will reflect characteristics of the total student population. It is also possible to test a larger group and have the sample chosen by computer to reflect percentages in the total population.

4. Order or have printed the required number of instruments, answer sheets, and ER forms needed to accommodate the number of respondents to be included in the sample size.

Discussion

Testing procedures will vary, depending upon the number of people in the sample size. The team needs to set up its testing procedures so that testing space can be scheduled and whatever tasks the team wishes implementers to oversee in regard to contacting and scheduling respondents can be incorporated into the team's training for implementers.

Mass testing has proven to be an efficient procedure for the model. When a mass testing approach is used, an adequate testing place must be located. If the sample is very large, then several mass testing sessions will have to be scheduled. However, the contacting and scheduling of respondents can be less complicated with this approach.

Letters inviting members of the sample to participate can have postcards enclosed that prospective respondents return, having indicated their willingness to participate and the testing session they will attend.

With the mass testing format, implementers could be used to follow up with respondent contact and scheduling for any additional sessions needed to complete the sample. Or the format could incorporate implementers visiting prospective respondents after the initial contact letter has been sent to personally explain the project, answer questions, and sign up respondents for the testing sessions.

If it is more appropriate to use smaller testing sessions, i.e., during class periods or at separate places on campus, then corresponding arrangements and tasks will have to be set up to accommodate this format. Implementers may have to assume a bigger role in respondent contact and scheduling procedures.

Another important aspect of the testing procedures can be arranging a reward for respondents. Respondents are being asked to volunteer their time and to respond to a thought-provoking and complicated assessment technique. Once they learn how their participation can benefit them through possible changes in the environment, they will recognize the reward for participating in the long run. If the team can offer an immediate reward for participation, such as a free pass to a recreational activity, this reinforces the process and hesitancies are more easily removed.

The team involves itself in these careful procedures in order to assure as close to a 100 percent response from the sample chosen. Questions of response bias (only certain kinds of students choosing to respond) must be dealt with when the response is below 100 percent. In practical terms, however, this will probably be a serious problem only when response falls below 85 percent of the sample. Careful and thorough planning in the on-campus applications of this model have yielded a response rate of close to 100 percent. Thus the team is advised to put efforts into procedures that will assure a high response rate in order to avoid having the interpretation of their results limited by questions of response bias.

Process

To set up its testing procedures, the team will need to:

1. Consider and choose the testing approach that will best accommodate its assessment technique and sample.
2. Locate and retain testing space.
3. Set up a method for contacting respondents.
4. Set up a method for scheduling respondents for the testing sessions.
5. Determine what role, if any, the implementers will need to assume in the contacting and scheduling of respondents.
6. Determine what reward, if any, the team can offer respondents. (Past planning teams have made arrangements and obtained free passes that respondents could use to attend a movie, game activity, or hobby session of their choice at the school's student union.)
7. Make appropriate assignments and schedule completion dates for the above tasks.

Discussion

During Stage I, the originator of the ecosystem project communicated the idea for the project and gained support from appropriate administrators, key agency staff members, and faculty. Later, as the team worked through Stages II and III, its members maintained liaison with some of these constituents as well.

The beginning of Stage IV provides an excellent opportunity to again bring these persons up to date on the project. This is important, since administrators and agency staff will have to be involved in considering the data analysis and in planning and implementing design projects. In addition, the team may wish to include such persons as implementers for the project. For instance, if assessment is to be done during class periods, faculty who teach selected classes will have to be contacted and asked to cooperate with the project. This usually means gaining support from key academic administrators prior to contacting individual faculty members.

Process

To enhance communication with key constituents, the team needs to:

1. Decide which persons and groups should receive information on the process at this time.

2. Determine the way to present this information. Two methods commonly used are a talk between a team member and key constituent or a presentation to a group by one or more team members. Another method, especially appropriate for an agency staff that is central to the project, is to briefly explain the model and then have the staff do a limited simulation of the model on its own agency environment. For an example of this process, see Technical Appendix E, p. 121.

3. Determine which task or tasks each team member will assume in this process.

Discussion

Staff that work within the environment or students who could receive academic credit for work on the project are prime candidates to be implementers. The number of implementers the team will need is dependent upon the size of the sample and the testing procedures that have been chosen. A general rule is to have one implementer for every 50 respondents so that there will be adequate personnel to assist with the ER analysis. A testing procedure that uses a number of small testing sessions will require enough implementers to administer each session without overscheduling implementers. A few additional implementers should be recruited and trained as stand-bys in the event of last-minute absences.

Training of implementers will depend upon the number of tasks the team wants them to assume in addition to administering the assessment technique and assisting with the ER analysis. Training needs to be carefully outlined, materials to be used need to be prepared and ready for handout, and training sessions need to be scheduled appropriately in advance of the first task that the implementers will be expected to conduct. On the average, training of implementers takes six to eight hours and can be easily designed to fit into several meeting sessions. If the implementers will not be responsible for scoring ERs, the training time is considerably shortened.

It is advisable for each planning team member to assume leadership of a group of implementers along the lines of the concept of captain and team. This will help the planning team to coordinate implementer activities and provide implementers with a ready contact and information resource when needed.

Process/Recruitment

To recruit implementers for the project's assessment and analysis, the team needs to:

1. Determine how many implementers will be necessary.

2. Determine whether students and/or faculty and staff should be recruit.

3. If students are to be recruited, contact probably will need to be made with those faculty members who have students for whom work on the project could constitute credit as a practicum or independent study. Likely faculty members to contact would be those in sociology, psychology, and education departments.

4. A team member should have a meeting with the prospective students and their faculty members in order to describe the project and its innovative research features and to explain exactly what the implementers will be trained for and expected to do. Without this preparation, misunderstandings are likely to arise.

5. If faculty and staff are to serve as implementers, a meeting with them to explain the project and the benefits expected from it that could ease their workload will become instrumental in recruiting them. Or if a new staff is about to be selected, such as resident hall assistants or advisers, the implementer role should be incorporated into the job description so the new staff will know that they are expected to assist with the project.

6. Provide recruits with an assessment and analysis schedule and the times and places that training will be held.

Process/Training

To design and give its training for implementers, the team or its member(s) responsible for training implementers may wish to review the basic training methodology given in Technical Appendix F. p. 127. Suggested guides for developing training sessions for implementers include:

1. Identification of each task the implementer will be expected to conduct.
2. The design of an appropriate training module for each task or group of tasks.
3. The incorporation of these modules into an overall format for the training schedule. (An example format for training ecosystem implementers is given in Technical Appendix G. p. 129.)
4. Prepare the needed training materials for each training sessions.
5. Implement the training schedule.

Discussion

The implementation of its assessment is an opportune time for the team to seek publicity. Until now, the team has kept its immediate constituents apprised of its activities and has concentrated on planning. Now the team is ready to move into action and has its plans formed well enough to issue anticipatory statements about the project and its goals, all of which make good material for publicity.

The publicity is needed to make the general campus community aware of the project and prospective respondents aware that their participation will be valuable to the subsequent design of a better environment. This not only provides an assist for the assessment's upcoming implementation but also begins to lay the groundwork for the team's future design work.

The student newspaper and campus radio/television station, if such exist, are important channels for project publicity. With some suggestions and guidance from the team, a feature story or series of articles about the project, its intent, and its team members could easily be negotiated. Student service or departmental newsletters should also be tapped. If the environment has personnel who have not become directly involved with the project, then it is advisable to make a special effort to inform them about the project so they can be supportive rather than suspicious about the assessment.

Process

Suggestions that the team may find helpful in launching its project publicity include:

1. Identifying all the publicity resources that the team could tap.
2. Contacting these resources to request publicity for the project.
3. If requested, assisting with interview arrangements or preparing copy about the project.
4. If team members consider it desirable, requesting opportunity to review a source's publicity about the project before publication.

Discussion

The team should allow sufficient time before testing sessions in which to contact and schedule respondents. The initial contact needs to establish among prospective respondents an interest in participation. It should include a description of the project and its goals, as well as the amount of time the participation will involve and how important the respondent's participation is to the success of the project.

It is often advisable to leave time in the schedule for follow-up contacts with prospective respondents. The goal, obviously, is to sign up and test as many respondents as possible. Follow-up measures can reinforce the team's initial recruiting efforts. Contingency plans should also be made in the schedule for additional testing sessions in the event that attendance at scheduled testing sessions falls short of expectation.

If assessment is to take place during class periods, the team will have to decide whether to inform students about the project ahead of time or at the start of the class period. Greater participation will probably occur if students are informed at the start of the class period, when they are already present. However, implementers must be trained in how to handle a situation in which a student chooses not to participate in the assessment.

Process

Guidelines for making contact and scheduling prospective respondents include:

1. The initial contact, whether achieved by a personal one-to-one meeting, group meetings, or phone or letter campaign, should describe the project and emphasize the unique potential it holds for environmental designs that can better serve the respondents.

2. The initial contact should also include an explanation of how the respondent was selected, how much time will be involved, how participation might ultimately benefit respondents with an improved environment, and that results of the testing will be available to the participants. For example, an initial contact with prospective respondents for an assessment of a residence hall environment might use an approach and content similar to this:

The school is using a new approach to gather information about how students view their residence hall environment. The new approach involves learning what you like and dislike about your environment so we can design a better residence hall environment.

Among the 4,500 students living in residence halls, a random sample of only 900 students has been chosen to participate in this innovative approach. You have been selected as one of these 900 students. The information you provide as a member of this sample group will remain anonymous. It will be used as background data for recommending changes in the environment that can better accommodate student needs.

Obviously, your participation is crucial if we are to obtain sufficient information to plan the needed environmental designs. We hope you will be willing to take about 90 minutes of your time to supply us this information. Special questionnaires have been devised to help you give us your views on residence hall life. Your reward is a chance to make the school more aware of your changing needs and enable it to plan accordingly. The results of the questionnaire will be available to all who participate in the sample group.

3. The initial contact should conclude with notification about testing sessions and how to sign up for participation. Depending upon the team's scheduling procedures, this may involve instructions for completing and returning an enclosed postcard or information about a second contact by an implementer to answer questions and sign up respondents, or specify sign-up locations and times.

4. Implement scheduling procedures and employ team captains to coordinate and oversee any follow-up contacts used in these procedures.

5. Prepare and use procedures to follow up initial contacts, should first recruiting efforts fall short of the sample's desired number.

6. Prepare procedures and schedules for additional testing sessions in the event these become needed.

Discussion

The team's biggest job during the assessment testing sessions is to make certain all procedures are carried out according to schedule and that everything runs smoothly. If a team concept is used in conjunction with implementers, then much of the coordination of testing activities and a final review procedures can be handled through the implementers' team meetings. Either team captains or one planning team member should be on call during the actual testing periods to resolve any unforeseen problems that might occur.

Process

It is often helpful if the team develops a checklist of details that need to be attended to in connection with the testing sessions. This check list might include items such as:

1. The correct quantity of instruments, answer sheets, and ER forms are ready for distribution to each testing session.

2. The testing place is properly equipped with everything necessary for taking the assessment, such as desks or tables, chairs, sufficient lighting, pencils, etc.

3. Any rewards the team has arranged to give respondents are ready and available for distribution.

4. Lists indicating which respondents are scheduled for which testing sessions are prepared and ready for distribution to implementers.

5. Each team member and/or implementer knows his or her assignments and schedules and has all materials needed for the testing session.

6. Assign and review back-up procedures for handling problems that might occur during testing periods.

7. The forms or scoring sheets which have been developed to facilitate the ER data analysis are ready for distribution.

In order to facilitate computer analysis of the instrument's data, the team will need to collect the instruments and answer forms when applicable from those who implemented the testing sessions and process the answer sheets for the computer center. The team member who has taken responsibility for liaison with the computer operators will need to keep in contact with their activities during this period.

Discussion

While the phase one instrument's data is being scored on the computer, the team may choose to begin scoring and analyzing the ER data that has been collected. Team members often find the analysis of ER data provides a vivid description of the environment and that they can document those things which are appreciated and working well--to a degree even team members might not have suspected.

And while those things in the environment that are not appreciated or that are considered detrimental in the environment will receive as much emphasis, the impact is softened by concrete examples of what might be done to change or improve the condition. When other types of assessments uncover or substantiate undesirable conditions, they often leave an air of depression because no one is sure what to do about it. On the other hand, an ecosystem assessment is more often a positive experience because ideas for improving the environment are suggested. And many of these ideas will not require great effort or massive redesign to enact. This gives the team a sense of accomplishment about the assessment and builds enthusiasm for the model's last stage in which many of these ideas become reality and are in turn evaluated.

Process

To begin the ER data analysis, the team implements its procedures and methods for scoring the information provided by respondents on the assessment's ER form. If the analysis procedure given in Stage III is used, then the sequence of activity would include:

1. Implementers taking an ER item count on the ER forms they have collected from the testing sessions.

2. Meeting of implementers with team captains to turn in ER forms and accompanying ER item counts.

3. Meeting of the planning team to develop an ER item chart based on the ER item counts.

4. Meeting to select which ER items will receive an ER content analysis.

5. Meeting of team captains with implementers to assign and distribute the ER forms for items to be analyzed. Review of ER analysis procedures and analysis sheets on which to record the information.

6. Meeting of implementers with team captains to return ER forms and analysis sheets.

7. Review of implementers' analyses by team captains to make any additional comments they feel necessary.

8. Meeting of the planning team to review ER content analyses. A common occurrence at this point is for team members to take pertinent ER suggestions back to their parent services and departments for staff review and the implementation of suggestions readily accomplished. An ecosystem assessment can provide a number of ideas that can and should be readily acted upon.

Discussion

When computer printouts on the instrument's data are received by the team, the team will have to spend some time deciding the most productive way to compare this information with the ER data. There will be some comparative analyses it will want to conduct and there will also be some comparative analyses that the individual services and departments represented on the team will want to conduct. The team will need to set up a schedule so that the data can be available for each purpose. In addition, the planning team will find that it often receives requests for the data from other groups and offices on campus. The team will then be faced with some policy decisions on who else may have access to the data when and for what purposes. The goal is always to share the data as widely as possible, but the team must decide how best to share the data results so that it will not be misinterpreted.

When individual services and departments analyze the information, they will develop methods of analysis which best meet their needs. The team as a whole may wish to adopt one of these methods or devise its own. In any event there will be no scarcity of information to analyze. In fact, the team will need to set some parameters for its initial analyses. Setting parameters is essential so that the team can begin to make preliminary reports back to respondents, to the campus, and to those who sanctioned and endorsed the ecosystem project. Much time will have elapsed since the team first was established and started its work, so that it cannot afford to wait until a full and detailed analysis has been completed on each piece of data. It will be readily apparent that this could take months. While a full analysis would produce a mountain of fascinating material, it would ill serve both the model and the planning team because it would delay start on environmental designs that could be applied and tested while the environment's population remained relatively the same and could seriously diminish the project's level of continuing support.

This hazard can be overcome by addressing the analysis of data in cycles or phases. The first round of analysis should, if possible, result in designs that can be implemented and evaluated on the same population. A second round of analysis utilizing additional data can then be conducted and designs implemented. The planning team can pursue this approach as long as it believes that the analysis is useful and can document its value.

The assessment technique's provision for an importance measure is a good guide in setting parameters for the team's initial comparative analysis. The team can choose five to ten of those items that the respondents indicated were most important as the extent of items on which an initial comparative analysis will be conducted. Items of special importance to individual team members and their constituents can be turned over to them for analysis. The information gathered through their efforts can then be fed back to the team. Meanwhile the team can concentrate its comparative analysis on those items that are most important to the respondents in preparation for environmental design and implementation.

Process

Guidelines helpful in conducting an initial comparative analyses of data would include:

1. The determination of items to receive comparative analysis for design purposes.
2. The establishment of policies for data use.
3. A schedule for data distribution to team member services and departments.
4. The formulation of a method to be used in the initial comparative analyses. Guidelines for an initial analysis might include the following:

a. Compare phase one instrument data and ER content analysis for items chosen.
b. Summarize this comparison.
c. Compare this summary to data from the phase one instrument which deals with closely related items (same scale). If there has been an ER analysis of two or more items from the same scale, this information can also be summarized and compared.

5. Set a date for completion of the initial analysis.
6. Make assignments for the analysis of items. The team, as a whole, may want to conduct the analysis, in which case it is advisable to schedule which items will be analyzed at which meetings, or each team member may select a group of items to analyze.

Discussion

Upon the completion of the data's initial analysis, the team needs to report its findings. This report can be qualified as preliminary to indicate more analyses are to follow. But it is essential that the team's respondents and campus constituents begin to see results from their participation and support.

The initial report should highlight important findings that the analysis has revealed. It is also desirable to include examples of changes that individual services and departments have put into effect or are planning to put into effect as a result of the ER analysis. This can enhance the status and credibility of both the team and the represented services and departments. The report should also indicate that the team will now move into its design phase for these findings.

Process

In reporting on the findings on its initial analysis, the team can:

1. Write a formal report for distribution to appropriate administrators including, of course, those who initially gave commitment and support to the project. It should contain a brief description of how the assessment was set up and should explain the unique advantage of its ER format. The use of underlining key content in highlighting the assessment's findings is always helpful to the reader. Follow-up reports can deal with specific areas in more detail.

2. Release a brief news report to each department or service resource the team used to publicize the assessment.

3. Make arrangements with the student newspaper and campus radio/television station to do a follow-up story or feature on the assessment reporting its findings.

4. Meet with or prepare an appropriate communication for respondents and implementers about the assessment's findings.

The team now has the information necessary to initiate Stage V design of the environment. It has started cultivating a receptive audience for its suggested redesigns, as well, through its efforts to inform the respondents and constituents of the assessment's initial finding.