Introduction

Evaluation and assessment are two very important aspects in the process of teaching and learning, especially within distance education. Dressel (1986) defines evaluation as "the collection and interpretation, through systematic and formal means, of relevant information which serves the basis for rational judgment in [learners'] decision situations" (p. 9). Assessment, on the other hand, is defined on what students and teachers consider important and meaningful and how they utilize their newly acquired knowledge and skills. Evaluation allows one to observe a learner's progress, as well as the success of educational programs in which they participate. Hanson et al. (1996) further indicate that "the purpose of student assessment, either through observable performance, product development, or traditional paper and pencil tests, is to provide data to the instructor indicating to what degree the performance objectives have been mastered" (p. 39). Evaluation is, therefore, very integral in the delivery of distance education.

Evaluation is an important social and cultural function. Lincoln and Guba (1989) have defined four phases of evaluation in America that demonstrate this importance. The first phase is defined as measurement goal-driven. "School tests had been utilized for hundreds of years, to determine whether students had 'mastered' the content of the various courses or subjects to which they were exposed" (p. 22). Dressel (1986) further notes that "the measurement model, with its emphasis on reliability, validity, and objectivity, emphasizes the uses of tests, norms, and standards and omits factors which are not operationally definable and objectively measurable" (p. 3).

The second phase, however, is defined as description goal-driven. Lincoln and Guba (1989) mention that "after World War I, it became evident that school curricula need[ed] to undergo dramatic revision, [to choose] an evaluation approach that… serve the purposes for evaluation now contemplated" (p. 27). This is perhaps the reason why the evaluators of the time tried "to determine effectiveness in the attainment of program objectives with careful advance planning, student performance, or other appropriate results provide feedback for diagnosing deficiencies and improving programs" (Dressel, 1986,p.3).

The third phase occurred in 1957, and was defined as judgment goal-driven. Lincoln & Guba (1989) state that "the call to include judgment in the act of evaluation marked the emergence of [a] third generation evaluation... in which evaluation was characterized by [the] efforts to reach judgments, and in which the evaluator assumed the role of judge, while retaining the earlier technical and descriptive functions "(p. 30). Dressel (1986) points out that this third phase produced quick evaluations that made "broad judgments rendered with minimal data collections" (p. 4).

The fourth phase -- which appears in the early 1980s -- is defined as an alternative approach, based on authentic assessments and context-based evaluations. This phase develops a constructivist approach to evaluation and assessment. Lincoln and Guba (1989) confirm this by stating that the phase was based on "a responsive constructivist evaluation" (p. 38). They also add, "Fourth Generation evaluation is a form of evaluation in which the claims concerns, and issues of stakeholders serve as organizational foci (the basis for determining what information is needed), that is implemented within the methodological precepts of the constructivist inquiry paradigm" (Lincoln & Guba, 1989, p. 52). This form of evaluation has therefore become a critical feature of distance learning.

Evaluation and assessment are related to distance education because:

Philosophical Perspectives Behind Evaluation: Positivism and Constructivism

There are two major philosophical perspectives that support the theoretical and instrumental roots of evaluation -- Positivism and Constructivism. They have very strong implications for the development and interpretation of evaluation procedures. Each perspective has its own conceptual definition, characteristics, and tools.

  Positivism: The Objective Approach

Positivism is defined as the objective approach to evaluation within education. It also is identified as the scientific method that is based on reliable and objective data -- which is the product of measurable experiments, tests, and statistical procedures. Its conceptual assumptions are as follows:

• The learners' goal is to gain knowledge, as educators provide it.
• Reality is analyzable and decomposable.
• Learning consists of assimilating the objective reality.
• The role of education is to help student learn about the real world.
• Role of teachers is to interpret events for them (Duffy & Jonassen, 1992, p. 138).

 Constructivism: The Subjective Approach

Constructivism is defined as the subjective approach to evaluation. It is also identified as the qualitative method based on individual performances, such as context-based criterion-reference. Learning outcomes, therefore, are individual and personal constructions. Reality is a subjective construction, meaning that each individual will construct his or her own interpretation about what is real and what is not. It is goal-free, meaning that there are not learning objectives pre-established by instructors without consulting with the learners' needs. Reliability and validity, however, are not important because the purpose of evaluation is not to measure learning outcomes in terms of statistical figures (Lincoln & Guba, 1989). This is so "because the constructivist paradigm perceives learners as interpreting what they learn individually and their learning outcomes being different from one another, instructors must learn to implement alternative evaluation methods with their students" (Bludau, Maddox, Pounds, 1998). Constructivist philosophy, therefore, has the following evaluation criteria: goal-free evaluation; authentic assessment; judgments based on knowledge, experience, and context; and, multiple and multimodal perspectives; and socially constructed meaning (Duffy & Jonassen, 1992). These criteria make the use of constructivism important in distance education.

The following tables compare and contrast the differences between the positivist and the constructivist approaches. Table 1 compares and contrasts the general characteristics, and Table 2 focuses on methodological characteristics.

Since Constructivism and Positivism are the cornerstone philosophies of evaluation, they are very important to understand the unique contexts of different distance education environments. The following sections explore the characteristics evaluation has within distance education settings, such as: elementary education, secondary education, higher education, and business and professional training.

Evaluation in Elementary Education Environments

Two categories of evaluation exist within elementary education: traditional and alternative assessment. Because these two types often prompt a discussion of what is right and wrong, it is necessary to understand why some people value and employ one form, but not the other. This judgement call often involves a person's or school's philosophy of education. Understanding the differing philosophies of education creates an objective context in which to examine traditional and alternative assessment.

A philosophy of education must include: the stated purpose of education, an identity of the types of people who are worthy of an education, and examples of how teaching and learning behaviors best facilitate learning. Armstrong cites that four different types of philosophies exist: essentialist, perrenialist, progressivist, and social reconstructionist (11-12). His summarized definitions of each of these philosophies are displayed in the table below.

One or more of these philosophies prompt educators to choose either traditional or alternative assessment. Traditional evaluation methods include standardized tests and quizzes -- usually multiple choice formats -- which prompt the recall of factual knowledge. While people maintaining other educational philosophies might occasionally use this type of evaluation, it is more likely to be employed by those who have an essentialist or perrenialist philosophy of education. Educators of these schools of thought want students to learn content as it is, without modification or application. For this reason, the uses of traditional testing tools are deemed to be the most appropriate.

In contrast, proponents of progressive and reconstructivist philosophies employ more modern, alternative forms of assessment. Alternative or authentic assessment takes a more student-centered approach to evaluation and includes both classroom-based assessment portfolios and student-designed assessments. Magnan-Lev identifies four principles of authentic assessment (5):

1. learner-centered
2. part of the learning experience
3. multiple forms of assessment
4. feedback

Classroom-based assessment: This is an approach used to determine how well content is covered and to identify students' learning styles. Teachers consider students' prior knowledge and experiences, culture, and language differences, factors not usually considered in many traditional forms of evaluation.

Portfolio assessment: Teachers design activities resulting in student-made products, which are collected to make a portfolio for each student. The portfolio of work is then used as the product by which the students are evaluated.

Student-designed assessment: Here teachers implement evaluation activities designed by students.

Magnan-Lev also provides examples of assessment tools useful in alternative assessment (5):

• questionnaires to be completed by students, prompting them to evaluate their own learning and performance, provides for an opportunity to reflect on their learning experiences and efforts
• teacher or learner observation
• checklists to collect data identifying areas or activities necessary for successful evaluation
• different forms of student-made reports such as charts or skits, resulting in products that reflect student learning
• learning contracts outlining individual or group goals, used to guide student learning and involvement

Both traditional and alternative forms of assessments have advantages and disadvantages. A person's philosophy of education influences his or her perception of these.

Although differing educational philosophies exist, the intended purpose of each is to facilitate learning. "Alternative forms of assessment might best serve some of these purposes while more traditional forms could still serve others". Being mindful of this will help guide educators charged with the responsibility to select appropriate evaluation methods for elementary classrooms. Educators should also be mindful of the developmental processes required to relate evaluation at the elementary level to later experiences at the secondary level.

Evaluation in Secondary Education Environments

Evaluation within the secondary distance education environment contains many of the same features as evaluation in elementary education and higher education. Among the most important of these features, is the reliance upon the two major evaluation philosophies: positivism and constructivism. Positivism tends to be the dominant method of evaluation within secondary distance education, since several of the activities are either criterion referenced to national standards, or are modified versions of standard evaluation. (Riley, 1996) Constructivist evaluation, however, is not utilized to the same degree as positivist design. The reason behind this, is perhaps the fact that it is very hard to subjectively evaluate materials in a distance education course due to the lack of face-to-face learner instructor interaction. Programs that utilize portfolio means of evaluation, usually use rubrics to grade students' work (Martinsen 1998). This means that much of the evaluation found in secondary education telecommunication environments follows a well stated pattern of either positivist or constructivist evaluation design.

Since simplicity and clarity are very important in efficient distance education, many secondary distance education programs use evaluation methods that follow the positivist philosophy. Therefore, much of the evaluation in this environment, therefore, relies on different adaptations of criterion referenced evaluation methods. Authentic evaluation in the secondary distance education environment, can be accomplished in either traditional or a non-traditional ways. The traditional method utilizes a local facilitator who can either be a parent or an instructor. The role of the facilitator, is receive, administer, and process different teaching materials that are sent to him or her by the sponsoring institution. Therefore, the local facilitator plays a non-instructive role in the evaluation process (Choike, 1998). The non-traditional method of authentic assessment, however, totally relies on the direct interaction with the student. The student is responsible for directly answering online exams and quizzes that are given without the assistance of a facilitator. Students become solely responsible for turning in work to their coordinating institution. Institutions that use this method usually provide html pages that are interactive and can be sent to the institution via e-mail (Choike, 1998).

While methods of positivist evaluation design revolve around authentic evaluation, methods of constructivist evaluation are based around portfolio assessment. With portfolio assessment, students are allowed to explore the topic areas and create their own sense of the subject (Lemonic, 1995). Projects, essays, and other assignments are kept portfolios that are submitted and evaluated by the instructor as needed. Many of these portfolios are graded in a rubric method, to ensure that the projects and other assignments contained within them meet the either nationally recognized or instructor set standards. Portfolio based assessment is many times used in courses where the promotion of higher order thinking skills corresponds highly with the degree of difficulty of the course (Riley, 1996). Appraisal of portfolio based evaluation can either be conducted by the use of real or virtual portfolios. Real portfolios are either graded by the local facilitator as the course is in progress or, is usually mailed to the sponsoring institution at the end of the semester to determine if the student deserves credit for the course. Virtual portfolios utilize workgroup software (like BSCW or FirstClass) to create a place where assignments can be posted and graded by instructors at the institution (Martinsen, 1998).

Many examples of institutions that utilize positivist and constructivist evaluation designs can be found. However, two institutions in particular stand out in their overall design: Oklahoma State University and the Earthstone Academy. Oklahoma State University's distance education program utilizes primarily positivist methods. Not only does OSU utilize traditional methods of authentic assessment, but it also uses online assessment via html quizzes and exams. Many programs at Earthstone, however, utilize constructivist based evaluation methods. Many of Earthstone's distance education programs use portfolio based assessment to align student work with the standards that are set by national organizations and the institution itself (Choike, 1998; Martinsen, 1998).

For both positivist and constructivist evaluation, there are certain distinct advantages and disadvantages. Positivist evaluation, for instance, is easily implemented and graded. The use of criterion referenced testing allows assignments to be easily graded, and rapid feedback given to the learner. The main disadvantage to this, however, is the fact that criterion-referenced tests and quizzes demonstrate not all of the learner's abilities. Constructivist evaluation, on the other hand, has the main advantage that it is demonstrative of the learner's abilities and knowledge of the subject. The main disadvantage, however, is the constant struggle to keep track of the large amount of data produced by the students. The use, advantages, and disadvantages of positivist and constructivist evaluation in secondary distance education closely represents similar patterns in elementary and higher education.

Evaluation in Higher Education Environments

Evaluation in higher education distance learning environments poses a unique challenge. This challenge is the result of the unique synchronous/asynchronous interaction between learners and their teacher. Although interaction at a distance is different from traditional face-to-face classroom interaction, evaluation tools used to assess learning outcomes are the same. Higher education evaluation is not exempt from the theoretical and practical implications of the two major perspectives already mentioned (Positivism and Constructivism). Therefore, both perspectives are used for selecting evaluation criteria and tools.

In higher education, distance educators have developed evaluation based on areas of course content, student performance, and teacher instruction methods (Holmberg, 1989). For example, Moore (1997) states that in higher education distance education programs: "evaluation was conducted on the basis of learners' academic performance and… attitudes and behaviors. Eight variables, including… attitudes scores… were analyzed to assess effectiveness" (p. 16). One of the major conclusions about the learners' attitudes and behaviors toward distance education instructional delivery is that:

• There is no evidence to support the idea that face-to-face instruction is the optimum delivery method,
• Instruction by teleconferencing can facilitate learning as well as or better than classroom instruction, and
• The absence of face-to-face contact is not detrimental to the learning process (Weingand, 1984, as cited by Moore, 1996, p. 16).

Conversely, course evaluations of distance learning programs have demonstrated their importance in the improvement of course content (Holmberg, 1989). Evaluations in courses are "designed to assess the broad or cumulative aspects of a student's educational development" (Dressel, 1986, p. 208). Distance educators have, therefore, concluded that: "course evaluation includes both the description of what works and the search for causal relations... to establish the cause of differences" (Holmberg, 1989, p. 194). These characteristics are very important for designing ad hoc and unique evaluation criteria and tools for distance learning. An important element to reference during the evaluation of distance learning programs and courses is the environment and context in which distance learning takes place. The relationship between students, the teacher, and the environmental characteristics is one of the important aspects of environmental assessment in distance education (Moore, 1997; Dressel, 1986).

Another aspect of higher education evaluation in distance learning environments is the use of formative and summative evaluations. Formative and summative evaluations in distance learning have proved to be very effective, since:

• Conducting a thorough needs assessment.
• Studying the context in which the project will operate.
• Analyzing policy and procedural alternatives.
• Monitoring all major activities and processes of both development and implementation.
• Determining the success of major components, programs, and systems.
• Providing continuous input to the design team.
• Identifying problem areas. (p. 35).

• Determining the success of major components, program and systems.
• Determining the success of the project as a whole by measuring student completion rates, achievement gains, and satisfaction.
• Measuring reactions and opinions of teachers, administrators, and the community in which the project is taking place.
• Measuring side effects and unanticipated outcomes.
• Determining desirable modifications for the next development cycle of the project or for the next project.
• Suggesting appropriate next steps (p. 35).

Evaluation in Business & Professional Environments

Evaluation in the business and professional environment emphasizes training and development as the means of learning. Therefore, evaluation is characteristically different from other areas of education. From a training and development perspective, instruction should enhance individual competence and performance (Edwards & Ewen, 1995). Instructional delivery methods must meet the needs, demands, expectations, and circumstances of each individual organization. To be fully utilized and appreciated, any evaluation of instructional delivery methods must be integrated into the goals, operations, and culture of an organization (Moseley & Solomon, 1997). One must know why s/he is evaluating, what s/he is evaluating, and how the data will be used (Piskurich, 1997). Evaluation methods should be determined based by the purpose of the specific project, and should meet the demands of the various stakeholders involved (Boyle, 1997).

The more traditional methods of evaluation include formative, summative, constructivist, and confirmative. Traditional evaluation methods often have limited utility in a business environment due to their historical nature. These methods rarely offer enough information to enable management to conduct a Return on Investment (ROI) analysis. Also, when used individually, they quite often fall short of meeting the complex needs of the business community. In short, evaluation should be a process, not an event (Lincoln & Dunet, 1995).

In a business environment, evaluation of instructional delivery methods should be focused on the actual development of competencies identified as critical to the organization's success. Prior to selecting an evaluation method or methods, the evaluation criteria must be established. In establishing these criteria, it may be beneficial to think of organizations as social systems -- which consist of relationships between departments and individuals. In order to function sufficiently, it is essential that the training and development department communicates and cooperates with upper management to develop effective training systems. It is through this communication and cooperation that these departments will develop a mutual understanding of the organization's learning objectives. Once this understanding is achieved, evaluation will lead to construction of shared expectations concerning exactly what objectives the training will address. For training to be effective, both the instructional designers and management must fully understand what the others expect to accomplish (Piskurich, 1997).

There are many different aspects of an organization that impact evaluation. For instance, if an organization's culture lacks appreciation for quality products and processes, the organization is likely to lack the understanding necessary to truly appreciate the value of evaluation. Without appreciation, it will be difficult to conduct evaluation in a thorough and effective manner (Rohrer-Murphy, et al, 1997).

Another important organizational aspect to consider is the identification of stakeholders and their need for evaluation information. Every organization has a multiple number of stakeholders, and not everyone within an organization has the same information needs. Some guiding questions that may be beneficial in identifying the needs of various stakeholders include: "How would you define success for the project?" and "What results would convince you that the project is worth doing?" (Lincoln & Dunet, 1995).  To put this concept into context, consider the various possibilities for needed information within an organization. For instance, the finance department may ask, "Will this training ultimately effect the organization's profit margin?" The legal department may ask, "Will this training minimize certain risk factors?" The production team may ask, "Will this training increase overall productivity?" Management may need information from all of these areas to support decision-making concerning the effectiveness of training methods (Lincoln & Dunet, 1995). Therefore, it is important that evaluation is flexible enough to tailor information to meet a wide variety of needs. By meeting the needs of stakeholders, support will be increased for training programs (Lincoln & Dunet, 1995).

Evaluation must also provide information to the training and development department and those responsible for instructional design. Evaluation is also the method by which the training department is able to justify the expense of training -- and perhaps even the existence of their department (Rohrer-Murphy, et al, 1997). Data concerning the overall effectiveness of training procedures, appropriateness of media and instruction methods, and other issues relating to possible revisions in instructional design may prove to be very valuable to training departments. This will assist them in better serving the needs of the other stakeholders involved. Also, the problem may not be solvable solely through the use of training. To prevent misuse of resources, information to this effect also must be collected and reported to management (Boyle, 1997; Lincoln & Dunet, 1995).

As mentioned above, each organization has a multitude of stakeholders whose needs should be addressed. Employees are among the stakeholders in an organization. As such, their development needs may contribute to the establishment of evaluation criteria. The employees are often more concerned with the additive effects of training. Employees may ask "Will this training help me do my job?" and "Will this benefit me?" (Boyle, 1997)

Many believe that Multi-Source Assessment (MSA) is a good tool for identifying employee development needs. In general, the most valuable employee performance information is situation specific --- this helps identify segments of performance that may benefit from training. MSA involves the gathering and reporting of information concerning employee performance, as well as, developmental needs from multiple sources. This in turn capitalizes on the availability of the collective knowledge about a specific individual. It is not intended to replace single-source assessment, but is offered as a means of supplementing traditional modes of assessment. Due to the credibility and multiplicity of information sources, MSA offers employees the opportunity not only to understand how their performance is perceived by their co-workers, but it also aids employees in understanding their own strengths and weaknesses. Information supplied by multiple sources is difficult to refute. It is, therefore, through the identification of an employee's various strengths and weaknesses that one is able to identify and target certain areas in need of development. As Edwards & Ewen (1995) so aptly stated, "MSA offers a unique opportunity to change employee training from a generic cafeteria style to a targeted approach based on defined needs from behavior feedback." (Edwards & Ewen, 1995).

• Enhances the quality of information
• Provides specific performance feedback
• Targets developmental areas
• Facilitates performance
• Allows measurement of training effectiveness
• Enhances self-knowledge and motivation
• Supports continuous learning
• Improves the reliability and validity of performance information (Edwards & Ewen, 1995).

• Data supplied to management should also support other business-related aspects as well. Data needs supplied to management should support quality assurance. In some instances, opportunities for quality checks and guiding improvement may also be built into the training program (Lincoln & Dunet, 1995; Rohrer-Murphy, et al, 1997).
• Data should also reflect changes in work performance and behavior. Evaluation provides a prime opportunity for measuring changes in work performance that may eventually lead to positive or negative business results (Lincoln & Dunet, 1995; Rohrer-Murphy, et al, 1997).
• In addition to its practical effectiveness, data should provide information regarding the cost-effectiveness and overall profitability of the training program (Lincoln & Dunet, 1995; Rohrer-Murphy, et al, 1997).
• Data should be predictive in nature. Organizations benefit from operating in a proactive mode as opposed to a reactive mode; therefore, data should assist management in identifying problems and predicting likely outcomes (Lincoln & Dunet, 1995; Rohrer-Murphy, et al, 1997).

Observation, an evaluation method commonly utilized in the workplace, may be informal or highly structured. Informal observation is often used to "conduct system acceptability," which is often a general check of training structure. Alternatively, highly structured observation often focuses on the individual performance of the user and commonly involves the utilization of a checklist of points to be observed relating to a particular task.

It is a common practice to supplement information obtained from observation with information obtained from interviews and focus groups. Interviews and focus groups provide a flexible tool for collecting data, by providing the interviewers with flexibility to follow up on questions. It is a method for obtaining data that is deemed relevant for the end-user. For this reason, this method may be extremely useful in determining the user's needs during formative evaluation. Interviews may be either structured or informal.

However, if interviews are are to be useful as a comparative tool between users, some structure will likely be necessary.  Questionnaires offer a structured tool that may provide information of a more quantitative nature and are often useful in conducting summative evaluations. Interviews and focus group discussions may, in turn, supplement questionnaires.

The advantages and disadvantages of the aforementioned evaluation methods are summarized in the following table:
 

Conclusion

Evaluation is the key to any learning process, especially in a distance education environment. Evaluation allows for the assessment of skills and competencies that are used throughout life. The evaluation philosophies prevalent in distance education today -- Positivism and Constructivism -- help clarify the process needed to understand how much information a person is actually retaining. Distance education evaluation at the elementary level is the first step of the process. At this level learners are just starting to form their ideas about reality and what they have learned. The implementation of evaluation at the secondary level further helps the learner to sharpen his/her skills on how to retain, process, and give more elaborate feedback. Distance education at this level becomes more complex as the learner matures. The use of Positivism and Constructivism is at its peak form at the higher education level. Here, distance education is the most challenging and difficult to retain and process.

Evaluation in business and professional settings is somewhat different, however. In these settings, job competencies and skills become the primary motivation behind evaluation. It is also at this level that evaluation becomes more important to persons other than the instructor and learner. In business and professional distance education environments, the knowledge and skills that the employees are able to retain greatly contribute to the work environment -- as well as the profit margins of the corporations utilizing the training. Although the differences are apparent between this level and its counterparts, the process of learning is just as important.

References

     Armstrong, D. G. (1989). Developing and documenting the curriculum. Needham Heights, MA: Allyn & Bacon.

 Bludau, H., Maddox, E.M., & Pounds, S. (1998).  Constructivism evaluation in telecommunications learning environments. In Applications of Telecommunications in Education.  [WWW Document].  Available:  http://www.coe.tamu.edu/~kmurphy/classes/telecom98s/constructivist/index.htm

     Boyle, T. (1997). Design for multimedia learning. London: Prentice Hall.

     Brown, G., Bull, J., & Pendlebury, M. (1997). Assessing student learning in higher education. New York: Routledge.

 Choike, Jim (1998). AP Calculus by Satellite Syllabus. [WWW Document].
Available:
http://www.cas.okstate.edu/extension/events/asts/calculus/index.html

     Dressel, P. L. (1986). Handbook of academic evaluation. San Francisco: Jossey-Bass.

     Duffy, T. M., & Jonassen, D. H. (1992). Constructivism and the technology of instruction: A conversation. Hillsdale, NJ: Lawrence Erlbaum Associates.

     Edwards, M. R., & Ewen, A. J. (1995). Making better performance easier with multi-source assessment. Performance and Instruction, 34(2), 20-22.

     Hanson, D., Maushak, N. J., Schlosser, C. A., Anderson, M. L., Sorensen, C., & Simonson, M. (1996). Distance education: Review of the literature. (2nd ed.). Iowa State University: Research Institute for Studies in Education.

     Holmberg, B. (1989). Theory and practice of distance education. New York: Routledge.

 Lemonic, Michael (1995). High-Tech High. [WWW Document]. Available:
http://www.pbs.org/newshour/bb/cyberspace/cyberspace_12-27a.html

      Lincoln, R. E., & Dunet, D. O. (1995). Training efficiency and effectiveness model (TEEM). Performance and Instruction, 34(3), 40-47.

      Lincoln, Y. S., & Guba, E. G. (1981). Effective evaluation. San Francisco: Jossey-Bass.

      Lincoln, Y. S., & Guba, E. G. (1989). Fourth generation evaluation. Newbury Park: SAGE.

 Magnan-Lev, M. (1997) The connection between cooperative learning and authentic assessment. In Adventures in Assessment. (Vol. 10). [WWW Document].  Available: http://www.sabes.org/aia101.htm. 

 Martinsen, Mark (1998). The Earthstone Academy Approach. [WWW Document].
Available: http://www.inconnect.com/~earthstn/intro2.html

     Moore, M. G., & Thompson, M. (1997). The effects of distance learning. (Revised ed.). University Park, PA: The Pennsylvania State University.

     Moseley, J. L., & Solomon, D. L. (1997). Confirmative evaluation: A new paradigm for continuous improvement. Performance Improvement, 36(5), 12-16.

 Nachman, E. (No Date).  Portfolio Assessment in language arts. [WWW Document].  Available: http://www.teachersworkshop.com/twshop/portfoli.html.

     Piskurich, G. M. (1997). Re-evaluating evaluation. Performance Improvement, 36(8), 16-17.

 Porter, A. (1991). Videoseries, Schools that work:  Videoconference #4 alternatives for measuring performance. (NCREL, 1991) [WWW Document].  Available: http://www.ncrel.org/sdrs/areas/issues/methods/assment/as500.htm

 Riley, R. (1996). Benefits of Technology Use.[WWW Document]. Available:
http://www.ed.gov/Technology/Plan/NatTechPlan/benefits.html

     Rohrer-Murphy, L., Moller, L., & Benscoter, B. (1997). A performance technology approach to improving evaluation. Performance Improvement, 36(8), 10-15.

 Winking, D. (1997). Ensuring equity with alternative assessment. [WWW Document].  Available: http://www.ncrel.org/sdrs/areas/issues/methods/assment/as800.html.