Editor's Note: This concentrated research serves as a valuable assessment tool for the ACSC Distance Learning Program. The study also serves as a valuable model for other in-depth comprehensive evaluation studies of Distance Learning programs across the board.

Formative Evaluation of the ACSC
Distance Learning Program: A Status Study

Donald A. MacCuish

Abstract

Reeves (1992) states that formative evaluation is the systematic collection of information for the purpose of making informed decisions about designing and improving educational programs.  Gleason and Donahue (1996) elaborated on the necessity of formative evaluation by stating that educational institutions have an obligation to conduct formative evaluation of their educational programs.  To date our formative evaluation of our distance learning program has been sound, but not as comprehensive as I would prefer.  In this paper I discuss our most recent and most comprehensive analysis (formative evaluation) of the Air Command and Staff College (ACSC) Distance Learning (DL) Program.  Our methodology in this evaluation, in my opinion, was rather unique.  Essentially I looked at our program through the lens of accepted learning theory and the 'best practices' as documented in the literature.

Introduction

The Air Command and Staff College is a part of the Air University located at Maxwell AFB, AL.  It is the intermediate level service school for Professional Military Education in the Air Force.  Our resident student body of approximately 600 students is comprised mainly of active duty Commissioned Officers in the US Air Force.  There are, however, students from the US Army, Navy, and Marine Corps as well as the Air Force Reserve, Air Guard, a limited number of civilians, and a contingent of International Officers representing 70 countries.

Curriculum content is determined by a number of factors, which include Congressional mandate, Joint Professional Military Education Requirements developed by the Chairman of the Joint Chiefs of Staff, needs of the Air Force, Air University's Continuum of Professional Military Education, etc.  Although these factors may appear to limit curriculum content, such is not the case.  ACSC has a great deal of latitude with regard to curriculum content.

The resident program is ten months long.  Upon satisfactory completion of the program students receive a diploma and Phase I Joint PME credit.  Those who maintain a 3.0 GPA are also awarded a Masters Degree in Military Operational Art and Science.

It is important that one have a solid understanding of the resident program curriculum because the curriculum of the distance learning program parallels it.  The major difference between the two programs rests on the mode of delivery of the instruction and the intensity of the learning experience itself.  Our distance learning program is much more reliant on technology than is the resident program.  The distance learning program averages 8,500 students who are at varying stages of completion, thus the student to student and student to faculty interaction is less intense.  We have mitigated these differences by providing our students a well-designed educational program, a good mix of media, and an aggressive approach to courseware improvement.

Methodology

As noted previously, the Distance Learning Directorate has an established evaluation program that incorporates all four levels of evaluation.  Based on our analytic efforts the data indicates that the ACSC Distance Learning curriculum is sound and properly represents the resident program.  The data also show that the program content is relevant to the needs of the nation, the Air Force, and our nonresident students.  Furthermore, the data support the notion that both instruction and learning are more than adequate.  We also know ACSC graduates of both the resident and distance learning programs make a positive impact in their duty assignments.  Our conclusions have been confirmed by outside agencies (for example the Air Education and Training Command's Inspector General, CINCs, Air University Board of Visitors, paid consultants, and others).

Even though we are confident in our successes and the strength of our distance learning program there is room for improvement.  The question is where should we focus our attention, more specifically our evaluation efforts?

Twenty years of experience in the design and development of training and education programs have convinced me that little is done in the area of evaluation of application of pedagogical theory and best practices.  Neither training nor educational program developers consult the literature as they are designing and developing their programs.  They tend to rely on experience, sixth sense, and established protocols.  Thus, we tend to believe that our programs are grounded in theory and best practices derived from quality research.  These two issues influenced my decision on the necessity of evaluating our performance in these two areas of our distance learning program.

The literature overwhelmingly supports the notion of program evaluation!  Saylor and his associates, for example, state that evaluation of an entire program is very important and that the goal of such an evaluation is to place a value judgement on all the educational experiences (Saylor, et al, 1981).  McNeil (1990) states that "Technologists evaluate a curriculum project by assessing (a) the merits of its goals, (b) the quality of its plans, (c) the extent to which the plans can be carried out, and (d) the value of the outcomes."  Pratt (1994) like Saylor, et al and McNeil emphasized evaluation during the planning, design, and development of an educational program, but had little to say about evaluation of a program already in place other than does it meet expectations.  In my opinion there must be a better way to evaluate a program other than to simply make an arbitrary value judgement about the program or assign a rating to each of the four McNeil criteria.

I wanted our evaluation effort to be quite specific with regard to what we do, how well we do it, and where we need to improve.  In my opinion abstract outcomes were not going to be acceptable.

Since there did not seem to be a literature base in the area of program evaluation to assist me in this effort I turned my attention to the types of research that might guide this research effort.  Several types of research, for example (experimental, ethnographic, and historical) were immediately dismissed because they, obviously, were not appropriate.  My consultation of Best and Kahn (1998) led me to the conclusion that the type of effort was descriptive, but the subcategories of descriptive research they identified did not seem particularly oriented to what I wanted to accomplish.  I also consulted Wiersma (1995) and Ary, Jacobs, and Razavieh (1996) in hopes that these authorities would be able to guide me in this effort.  I finally consulted Hopkins and Antes (1990) who provided me with some guidance.  Under their section on descriptive research they discussed what they called "The Status Study."  In the status study the researcher reports on the current conditions.  This report is then used to make administrative decisions concerning the specific program being investigated.  This methodology suited my needs.  Thus, I had finally been able to identify a research methodology suitable for evaluation of a complete educational program.

I looked at our program from two perspectives.  First was from the theoretical perspective.  Are we following good pedagogical theory?  If our program is not grounded in theory, then why not?  If this is the case what corrections must we make?  If our program is grounded in theory, do we know what theory we have adopted, why we have adopted it, and are they the most appropriate theories to use?  The other perspective is what I call best practices.  I wanted to know whether or not we were appropriately matching best practice to the situation or the circumstances.  I do not believe you can have an effective educational program if pedagogy and practice are not synchronized.

Admittedly, what I have included in this paper is not comprehensive, but rather representative of the findings rendered as part of this project.  I have included both good and bad findings in this article so it can be a valuable reference for the reader.

Theories

Theories and models are not simple academic abstractions designed to cloud the unknown, rather they are thoughtful philosophical or empirical ways to explain events, which are thought to occur or in fact do occur.  Models are one method we use to illustrate the execution of a theory.  I have chosen to begin my formative evaluation of  the ACSC distance learning program by addressing curriculum development.

Curriculum Development - In 1986 this researcher investigated the curriculum development process at the four war colleges (MacCuish, 1986a).  In this study I reminded the reader that  "Curriculum construction in the United States is generally conducted in a shockingly piece meal fashion (Zais, 1976)."  I mentioned this as one of my reasons for undertaking the study of the curriculum development process at the four War Colleges.  I used four models (academic, experimental, technical, and pragmatic) described by Gay (1980) as a basis for my investigation of the curriculum development process at the war colleges.  I was able to determine that the Zais statement noted above did not apply to any of the War Colleges.

Since the ACSC DL curriculum parallels that of the resident program, I thought it imperative to determine how the curriculum of the resident program was developed.  It was critical that I not make assumptions concerning development of the curriculum.  I referred, naturally, to my prior curriculum development study of the four War Colleges.  I am convinced the curriculum development processes used by both the Air War College and ACSC are based the academic model for the 'academic' content areas and the technical model for the areas considered as 'operational'.

Although this may seem odd, it is not.  Rather, it is quite appropriate.  The ACSC distance learning program, like the resident program is rooted in a two-semester system.  The first semester's courses are academic in nature (National and International Security Studies, Nature of War, and Air Power Studies).  Thus developing this portion of the curriculum using Gay's Academic Model makes a great deal of sense.  Using the "academic subjects" of the first semester as a knowledge base, the second semester courses (Aerospace Operations, Joint Forces, Campaign Planning, and the culminating Air Force Exercise) are technical courses on the application of aerospace power.  Here the Technical Model drove the curriculum development process.  Of particular interest is the Leadership and Command Course, which is divided into three phases.  It shows the influences of both the academic and technical models.

Thus, the ACSC curriculum development process is grounded in theory.  The two theoretical models ACSC has chosen to use are appropriate for the institution and the learning outcomes required by the nation.  Bruner (1977) best articulates another reason these two models are appropriate to the ACSC curriculum.  The ACSC curriculum provides the structural elements described by Bruner in his classic work The Process of Education.  ACSC has developed a "spiral curriculum" designed to educate the professional Air Force Officer with the knowledge and skills necessary to implement the National Security Strategy of the United States.  This is significant because the graduate of the ACSC distance learning program has the ability to integrate the military instrument of power with the other three national instruments of power (diplomatic, economic, and informational).

Course Design

American Psychology is dominated by behaviorism.  Because of this dominance the underlying principles of behaviorist thought predominates the field of training and to a lesser degree education.  There are a number of practitioners who do not care to acknowledge that there is a difference between training and education.  Woodington (Saylor, et al, 1981, p. 361) for example states that the intent of training is to enable people to gain new knowledge and skills in order to do their jobs, improve job performance, and develop specific competencies. Education, however, is "the acquisition of the art of the utilization on knowledge (Saylor, et al, 1981, p. 140).  Schreiber and Berge (1998, p. 179) assert that training "responds to organizational processes and job functions," while education "aims at enriching and expanding the role of an individual within a profession or society."  Dipboye and his associates (1994) state that the purpose of training is to better the performance of the organization by improving the job skills of its personnel.  Schreiber and Berge (1998, p. 414) also make the distinction between education and training noting that the focus of training is performance outcomes, while education is much broader in scope, specifically the acquisition of knowledge and how to apply that knowledge in novel ways. 

I think these few references adequately sustain my position that there is, at the minimum, a philosophical distinction between education and training.  This distinction, I suggest is so profound that it affects the basic assumptions made by the instructional developer and thence the philosophical approach taken by the instructor in designing the course as well as the instructional practices that will be utilized.  The importance of this distinction continues to manifest itself in the ACSC distance learning program.  The failure of the Air Force to recognize and appreciate the implications of the subtle distinctions between education and training is the major deficiency identified in this evaluation effort.

There are a number of learning theories, all of which can be grouped into categories or families.  For example, Hilgard and Bower identify eleven different categories, while McDonald identifies six, and Gage three (Knowles, et al, 1998, p. 20 and 21).  Kearsley (2000) provides an in-depth description of 50 learning theories on his web site.  Knowles, et al (1998, p. 22) try to simplify things for us by stating that "learning theories fall into two major families: behaviorist/connectionist theories and cognitive/gestalt theories, but not all theories belong to these two families."  The theories that are easily placed in the two major families are, obviously, the most widely used.  Herein is the problem at ACSC.

Several hundred thousand military and civilian personnel are enrolled in the education and training programs offered by the Air Force each year.  The overwhelming majority of this student population is enrolled in programs designed to develop and enhance job performance skills, e.g. training programs.  A learning theory from the behavioral/connectionist theory family is most appropriate if we intend to reach the desired learning outcomes of Air Force training programs.  The model the Air Force has selected is the Instructional Systems Design (ISD) Model (a.k.a. The Systems Approach to Training), whose roots are most commonly associated with such prominent dignitaries as Briggs, Cary, Deterline, Dick, Gagne, Merrill, and others.

Air Force Manual (AFMAN) 36-2234 (1979) describes the Instructional Systems Design process that is to be used for the development of all educational and training programs in the Air Force.  AFMAN 36-2236 (1994) is the official Air Force instructor's guide for the development of all instruction whether it be for education or training.  The problem with both of these publications is that they mandate a behavioral approach to both education and training.  I agree that behaviorism should be the foundation upon which training programs are built primarily because this approach targets objective and observable measures (Matlin, 1994).  I can not agree that a behavioral approach to education is correct primarily because behaviorism rejects "any terms referring to mental events, such as image, idea, or thought (Matlin, 1994, p. 5)."  Additionally, Gagne (1985) emphatically states that your theory of instruction must be appropriate for the desired learning outcomes.

Furthermore, I agree that the ISD process can be used as an instructional development umbrella for both training and education.  Historically, this has lead to at least two major problems with regard to PME.  First, training is process oriented, which usually means one correct solution that is reached in an orderly and sequential fashion.  Usually there are many 'correct' solutions to education problems.  Thus, the ISD process, if rigidly employed, limits the innovative application of knowledge.  If the process is not used in this manner, but only as a skeleton upon which we drape 'educational' models, processes, etc. then we ignore the elements of the model that are it strengths.  As a result we create problems for our students as described in my second objection.

Second, since the Air Force is a technical service, training is very important.  By the time Air Force Officers qualify for enrollment in the ACSC program they have already completed one or more training program, and most have served as a trainer in at least one previous assignment.  Thus, when they enroll in the ACSC DL program they tend to gravitate to a training mentality looking for processes, procedures, sequencing, and school solutions rather than the acquisition, assimilation, and application of knowledge in a fluid operational environment because they recognize the ISD features of the DL program.  It is for these two reasons that I am a strong advocate for the adoption of a different model for PME institutions.

I suggest that the ACSC Distance Learning directorate adopt a more traditional, cognitive model, for developing instruction rather than continuing to misuse the ISD Model as an umbrella upon which we drape our educational program.  My position is supported by the fact that a number of our instructional development assumptions are based on cognitive learning theories, specifically those of Argyris (1976), Ausubel (1978), Bruner (1977), DeBono (1971), Festinger (1957), Knowles (Knowles, et al, 1998), Piaget (adapted for adult learning situations, MacCuish, 1986b), and Spiro, et al (1988) among others.

Our educational philosophy and thus many of the learning strategies we incorporated into our program are based on adult learning (Knowles, et al, 1998; Kearsley, 2000).  The sequencing of instruction is rooted in theory.  For example Bruner's Constructivist Theory provides a general framework for the sequencing of instruction based on  cognition, the "spiral curriculum" (Bruner, 1977 and Kearsley, 2000).  The notion of learning based on the development of cognitive structures, most commonly referred to as schemata is well established and documented in Piaget's Genetic Epistemology (Piaget, 1977; Wadsworth 1972; MacCuish 1986b; Gruber & Voneche, 1977;  Kearsley, 2000).

Ausubel's Subsumption Theory shares conceptual frameworks with Bruner, Piaget and Spiro, et al.  This theory is appropriately used in the ACSC distance learning program.  Two aspects of his theory are significant.  First, he notes that it is oriented to individuals learning vast amounts of material from verbal (video) and textual materials (Kearsley, 2000).  All of our learning materials are relevant to the educational outcomes.  Additionally, we continually review these materials for relevancy, substance, and currency.  Second, Ausubel stresses the importance of "advanced organizers."  Advanced organizers help to explain, integrate, and show the interrelationship of the learning materials prior to presenting the actual learning material.  These are more than lesson overviews and summaries (Ausubel, 1978; and Ausubel, et al, 1978).  We incorporate advanced organizers in a number of ways, for example course and lesson thesis statements, lesson materials rationale, lesson introductions (video), OPMEP requirements, etc.

In our Leadership and Command Course, for example, we have incorporated a specific learning theory to help us improve the desired cognitive outcomes of each phase in the course.  Double Loop Learning is a theory of action and is quite appropriate in helping people learn to change their values and assumptions (Kearsley, 2000 and Hughes, et al, 1999).  The notion of double loop learning not only helps guide us in the development of our course materials and learning experiences, but also helps students better understand how to take charge of their own learning with regard to values and assumptions.  Once students understand this approach, we integrate it into the "Mentoring" Module of this course and encourage them to adopt this approach when mentoring subordinates.

We also use the principles of Festinger's Cognitive Dissonance Theory in the Leadership and Command Course.  If people are truly going to change their beliefs, opinions, and values then we must create 'cognitive dissonance' (Kearsley, 2000).  Whenever people talk about values, they want to know whose values are we going to teach.  Our goal is not to impose out values, but to challenge them with the idea that the Core Values of the Air Force (Integrity first, Service before self, and Excellence in all that we do) are really universal.  We accomplish this task by challenging basic assumptions.  We provide documentation to the students that highly respected researchers have proven that there is a universal set of values that transcends all cultures, religions and ethnic groups (Kidder, pp. 90 and 91, 1995).  We then show that the three Core Values of the Air Force summarize these.  It also helps us illustrate that all members of the Air Force share these values.

Spiro, et al (1988) articulated a theory of Cognitive Flexibility.  This theory emphasizes the ability to spontaneously restructure one's knowledge in response to rapidly changing situations (Kearsley, 2000).  DeBono in his theory of Lateral Thinking addresses issues associated with the generation of novel solutions to problems (Kearsley, 2000).  Both of these theories are relevant to today's Air Force officer in the employment of aerospace power in support of the National Military Strategy of the United States.  The principles of these two theories link our instructional practices.  The proof of our success is only realized in the successful conduct of air operations.  The failure of our efforts will be seen in the deaths of American, allied, and coalition military personnel.  In a worst case scenario it will be depicted in a catastrophic military defeat on the battlefield.

Best Practices

Reeves (2000) reiterated a comment made by Cronbach in 1975 about empirical research.  The essence of this comment is that "empirical research is doomed to failure because we simply cannot pile up generalizations fast enough to adapt our instructional treatments to the myriad of variables inherent in any given instance of instruction" (Reeves, 2000, p 5).   When we consider trying to apply empirical research to technology based learning situations the reality of this comment is even more profound.  Thus, we chose to evaluate our practices by studying what other practitioners have written and compared what we actually do to many of the suggestions made.  We evaluated our practices against some of these suggestions, but not all.  When we did not use a suggestion our decision not do to so was a conscious one.  Our goal was to determine whether or not there is agreement with what we do.

I began the evaluation of our  'Best Practices' by consulting Vincent Flanders' web page (http://www.webpagesthatsuck.com).  I found his comments and critiques to be quite useful.  Since the ACSC course materials are distributed both by CD-ROM and via the ACSC DL website, we thought it wise to enable students to transition from one media to the other as transparently as possible.  To accomplish this task we had to ensure that the organization of materials in the website and on the CD-ROM were similar.  When we compared our approach to the Flanders guidelines contained in his publication Web Pages that Suck (Flanders, 1998) we found our structure was 'right-on.'  For example, Flanders (1998) emphasizes the importance of a well-structured web site that facilitates navigation, is aesthetic, and minimizes the likelihood that students will get lost in cyberspace.  Our technical staff, after reading and studying the literature, incorporated sight identifiers on every web page to assist students in navigation and minimize the likelihood of becoming lost.  Golas (2000) in her comments on navigation included the ease with which a student can return to the starting point and a lack of deadends.

In my research I also found that our approach is considered good practice by DeBra (1996) who emphasized the importance of a clear and systematic organization of the learning site and Smith, et al. (1997) who contend that instructional content should be modular.  Modularization provides two distinct benefits to us as an organization.  First, it facilitates student navigation and reduces the likelihood of their getting lost in cyberspace, which was one of the concerns voiced by Flanders.  Second, when we have to update or revise our course materials it is easier and more efficient to pullout the old module and plug in the new one than to revise the whole course.  This approach is a matter of sound instructional practices and technological flexibility.

Although Schneiderman (1998) indicated that the depth of menus could go to 4 or 5 levels, we purposefully limited our depth to two and in rare instances three levels.  We thought that going to four levels was both unnecessary and could lead to student boredom just like answering services do after the third level. 

In another instance where our approach varied from one recommended by another practitioner is use of color.  Reemers (1998) recommends that you use no more than four colors per web page.  We determined that if our screen design was done well and we did not use color for the sake of having color, then we could exceed the four-color limitation.  Student feedback validates our decision.

Several other researchers provided insights about interface design guidelines that also helped us formatively evaluate the ACSC DL program as well.  Florida Gulf Coast University (2000) hosts its recommendations for the design of online instruction on the university's website.  We already follow many of the principles suggested in the FGCU recommendations.  We further compared our practices to those articulated by Orr, et al (1993) and Golas (2000).  Since our CD-ROM and online practices are in harmony with so many published recommendations we are very confident in our organization of the website as well as the CD-ROM.

Also of concern to us is the presentation of distance learning materials.  For example, Beltran (1996) recommended that the line length of text should not exceed 60 characters and should be in standard font sizes. With the exception of the readings, which were intended to be printed and not read on a monitor, we adhere to Beltran's principles as well. 

Our use of both CD-ROM and the website as storage areas for our readings presented us with some additional concerns.  Interestingly, several of our resident students evaluated Interactive Multimedia as an instructional tool in 1995 (Wilson, et al, 1995).  They empirically verified many issues related to screen design, text fonts, and standardization of lesson formats.  They noted that serif faces although not flashy or "full of character" cue the reader to quickly identify letters in a word.  They also verified that such fonts are more legible and each letter is more distinguishable than other faces.   As a matter of Air Force policy we are required to use serif faces so that was not an issue, unlike my earlier comments about our having to use the ISD process.  Last year we initiated a lesson standardization policy.  This intuitively made a great deal of sense. 

Previous to the 'standing-up' of the current Distance Learning Directorate this type of standardization was impractical.  Each of the three teaching departments had responsibility for both the resident and distance learning courses within their domain.  Each department had different standards, policies and methods.  As a result within each department resident and distance learning materials were standardized, but this same standardization did not exist across the distance learning program.  We believe many of our students would have been spared a great deal of confusion, which they articulated in their feedback forms, had the reorganization occurred earlier than it did.

As a direct result of reading the Golas and Wilson papers, I began to search for what others had written.  Nielson (1997) verified the wisdom of our decision and confirmed what Golas and Wilson, et al had written. Feedback from our students supports our decision.

In the mid 1990s, when the ACSC DL Course Materials were first moved from printed matter to CD-ROM/Web-based materials someone overlooked the strength and weaknesses of CD-ROM and the World Wide Web as instructional media.  Except for a few Asymmetric Toolbooks these two media served as simple storage systems for the readings.  It did not take long for ACSC to recognize this mistake.  Our mistake in this situation reminded us that different types of knowledge, course content, etc. require different forms of media (Gagne, 1985; Merrill, 1991; Merrill, et al., 1997, Merrill, 2000).  We have not forgotten this expensive lesson. 

As mentioned earlier, ACSC instituted its current organizational structure in January 1999.  Those of us who were instrumental in this reorganization effort quickly analyzed the presentation of course materials.  Prior to January 1999, the ACSC Distance Learning Program was limited to readings and Asymetrix ToolBook multimedia presentations.  Student feedback confirmed our initial reaction.  Since that time we have made significant improvements to our program.  The readings are fundamental to the education of midcareer officers, so our effort in this area was to eliminate redundant readings thereby making the reading more manageable.  We also rewrote the ToolBook exercises into html format.  Unfortunately, we have not found anything in the literature concerning these two items, so our evaluation is limited to student feedback, which does validate our approach.

Also in our preliminary evaluation of the distance learning program was concern about student knowledge.  We felt that readings and multimedia exercises were insufficient.  So we introduced synchronized multimedia using RealPlayer tools in the fall of 1999.  Golas (2000) stresses the benefits of this type of multimedia presentation.  The combination of student feedback and her comments are strong endorsements of our having implemented synchronized multimedia learning events.

This evaluation of the ACSC DL Program confirmed our greatest concern, student assessment.  At the present time we are limited to proctored multiple choice tests given at the Base Education Office.  This limitation is due to a number of factors that are, until now, difficult to overcome.  Our preference is to move to either an essay format or a scenario-based format or a combination of the two.  At this level, our objective is to not assess a student's knowledge, but his or her ability to organized, analyze, and present a solution to an abstract problem in a coherent manner.  Thus, multiple-choice items are not good indicators of student knowledge.  Students have also voiced their concerns in this matter because they recognize that the multiple-choice format does not prepare them to organize, analyze, and present a solution in a military staff environment.

To overcome this deficiency we are looking at two options.  The first is evaluation of a computer-enhanced assessment of essay materials.  This assessment format, although completely automated, promises to correct many of the limitations of essay examinations.  If proven successful, then we will be able to not only add a writing component to our distance learning program, but also have students recommend a course of action to their commander in the form of a staff study.  This means that we will not only assess student knowledge, but provide the student with a learning experience as well.

The other possibility is to construct an ongoing student assessment that ties each of the eight courses in the program together.  The key factor will be a computer-based scenario that requires student input at several times in each course.  This approach would be analogous to a war game.  War games have proven themselves as invaluable learning and assessment tools.

Conclusion

This paper describes a formative evaluation of the Air Command and Staff College's Distance Learning Program.  What separates this evaluation apart from others found in the literature is its comprehensiveness.  In addition to student feedback, which admittedly is important to us, we evaluated every aspect of our distance learning program.  We have evaluation documentation showing that the ACSC curriculum is well thought out and balanced.  We have verified that it serves the needs of the US Air Force.  We have determined that the approach to instruction and the learning activities we have incorporated are grounded in theory.

We carefully analyzed our educational practices.  Although they made sense to us, we wanted to determine whether or not our approaches were in agreement with those of other practitioners.  If they were, we wanted to know which ones.  If they were not, then we wanted to know why not.  Perhaps we are innovative.  Perhaps we had not thought of the methods used by someone else.  Perhaps we simply chose not to use those methods or perhaps the methods being used were not appropriate to our situation.

Our results in this regard were mixed.  In some instances we are 'right on' with what others are doing.  In some situations we are 'leading-the-pack'.  In others, we were simply wrong in our approach and the approach other practitioners have adopted have helped us improve our program.  And, in some instances their ways are not our ways and appropriately so. 

Perhaps the most important outcome of this evaluation, in this author's opinion, is the realization that the method or process we currently use to design and develop the lessons and course materials in the distance learning program represents a disservice to our students.  The reason for this is our students recognize the artifacts of the ISD Model and these overshadow the desired learning outcomes of the educational program.

The US Air Force, as do the other military services, requires the use of the Instructional Systems Design (ISD) or some variant thereof, for both education and training.  The ISD Model for instructional development is based on behaviorism and is quite appropriate for training or courses that require students to be 'able to do' something (Golas, 2000).  At the Air Command and Staff College we are concerned with the 'knowledge' that the student 'needs to know.'   We educate our students so each one can lead in developing, advancing, and applying aerospace power across the spectrum of service, joint, and combined military operations.  In my opinion, therefore, a cognitive model, not one rooted in behaviorism, is more appropriate especially when you consider the fact that we do not, in reality, adhere to the basic precepts of the model.

This comprehensive evaluation of the ACSC Distance Learning Program should be helpful too not only our sister service schools, but to colleges and universities that have distance learning programs too.   Without a complete analysis of your educational program how can you honestly and accurately determine its strengths and weaknesses?  A comprehensive evaluation process ought to include curriculum development, design of the instruction, learning theory, and best practices.  I suggest that most institutions severely limit the evaluation of their distance learning program to the technology used, comparison of the grades received by resident/distance learning students, student impressions of learning at a distance, and a few other factors.  I think we ought to do better than that.

References

Argyris, C (1976).  Increasing Leadership Effectiveness.  New York: Wiley

Ary, D., Jacobs, L.C., and Razavieh, A. (1996).  Introduction to Research in Education, 5^th ed. Fort Worth: Harcourt Brace College Publishers.

Ausubel, D. (1978).  "In defense of advanced organizers: A reply to critics," Review of Educational Research. 48, pp 251-257.

Ausubel, D., Novak, J. and Hanesian, H. (1978).  Educational Psychology: A Cognitive View. 2^nd ed.  New York: Holt, Rinehart and Winston.

Best, J.W. and Kahn, J.V. (1998).  Research in Education. 8^th ed. Boston: Allyn and Bacon.

Bruner, J. (1977).  The Process of Education.  Cambridge, MA: Harvard University Press.

DeBono, E. (1971).  Lateral Thinking for Management.  New York: McGraw-Hill.

DeBra, P.M. (1996)  Hypermedia structures and systems.  Web course, Eidenhoven University of Technology.  http://www.win.tue.nl/2L670/static . 

Department of the Air Force (1994).  "Guidebook for Air Force Instructors," AFMAN 36-2236.  Washington: US Government Printing Office.

Department of the Air Force (1979).  "Instructional Systems Development," AFMAN 36-2234.  Washington: US Government Printing Office.

Dipboye, R.L., Smith, C.S. and Howell, W.C. (1994).  Understanding Industrial and Organizational Psychology: An Integral Approach.  Fort Worth: Harcourt Brace College Publishers.

Festinger, L. (1957). A Theory of Cognitive Dissonance.  Stanford, CA: Stanford University Press.

Flanders, V. (2001).  Web Pages That Suck.  http://www.webpagesthatsuck.com (Downloaded on 25 January 2001).

Florida Gulf Coast University, Design Principles for Online Instruction, http://www.fgcu.edu/onlinedesign (Downloaded on 27 March 2000).

Gagne, R.M.  (1985).  The Conditions of Learning and Theory of Instruction, 4^th ed.  New York: Holt, Rinehart and Winston.

Gay, G. (1980).  "Conceptual Models of the Curriculum-Planning Process."  Considered Action in Curriculum Improvement.  Alexandria, VA: American Society for Curriculum Development.

Gleason, S.C. and Donahue, N. (1996).  "Boston revisits schools based management," Educational Leadership, 53, pp. 24-28.

Golas, K. (2000).  "Guidelines for Designing Online Learning," Interservice/Industry Training, Simulation and Education Conference Proceedings.  Washington: National Training Systems Association.

Gruber, H.J. and Voneche, J.J. (1977).  The Essential Piaget.  New York: Basic Books, Inc.

Hopkins, C.D. and Antes, R.L. (1990).  Educational Research: A Structure for Inquiry. 3^rd ed. Itasca, IL: F.E. Peacock Publishers, Inc.

Hughes, R.L., Ginnett, R.C. and Curphy, G.J. (1999).  Leadership: Enhancing the Lessons of Experience.  Boston: Irwin McGraw-Hill.

Kearsley, G. (2000).  Explorations in Learning and Instruction: The Theory into Practice Database.  http://www.gwu.edu/%7Etip/ (downloaded on 11 December 2000).

Kidder, R.M. (1995).  How Good People Make Tough Choices.  New York: Fireside Books.

Knowles, M.S., Holton III, E.F. and Swanson, R.A.  (1998).  The Adult Learner.  Houston: Gulf Publishing Company.

MacCuish, D.A. (1986a).  "An Investigation of the Curriculum Development Process at the Four War Colleges."  Orlando: Online Computer Systems, Inc.

MacCuish, D.A. (1986b).  "Let's talk theory: Piaget and the videodisk,"  Proceedings Society for Applied Learning Technology, Spring Conference.  Warranton, VA: Society of Applied Learning Technology.

Matlin, M.W. (1994). Cognition, 3^rd ed. Fort Worth: Harcourt Brace Publishers

McNeil, J.D. (1990).  Curriculum: A Comprehensive Introduction. 4^th ed. New York: HarperCollins Publishers.

Merrill, M.D. (1991).  "Constructivism and Instructional Design,"  Educational Technology, May, pp. 45-53.

Merrill M.D. (2000). Components of Instruction: Toward a Theoretical Tool for Instructional Design.  http://www.netg.com/research/merrillwp.htm (downloaded on 8 December 2000).

Merrill, M.D., Drake, L.D., Lacy, M.J., Pratt, J.A., and the ID2 Research Group (1996).  "Reclaiming Instructional Design,"  Educational Technology, vol. 36, no. 5, pp. 5-7.

Nielsen, J. (1997).  "Interface design for Sun's WWW site."  http://www.useit.com/alertbox/9711a.html (Downloaded on XXX).

Orr, K.L., Golas, K.C. & Yao, K. (1993).  "Storyboard Development for Interactive Multimedia Training," Interservice/Industry Training, Simulation and Education Conference Proceedings.  Washington: National Training Systems Association.

Piaget, J. (1977).  The Development of Thought Equilibration of Cognitive Structures.  New York: Viking Press.

Pratt,  D. (1994). Curriculum Planning: A Handbook for Professionals.  Fort Worth: Harcourt Brace College Publishers.

Reeves, T.C. (1992).  "Evaluating schools infused with technology, Education and Urban Society, 24, pp. 519-535.

Remmers, E. (1998).  "Guideleines for WWW-based Support Environments for Education Professionals."  http://users.edte.utwente.nl/remmerse/litstudy (Downloaded on XXX).

Saylor, J.G., Alexander, W.M., and Lewis, A.J. (1981).  Curriculum Planning for Better Teaching and Learning. 4^th ed. New York: Holt, Rinehart and Winston.

Schreiber, D.A. & Berge, Z.L. eds. (1998).  Distance Training: How innovative organizations are using technology to maximize learning and meet business objectives.  San Francisco: Jossey-Bass Publishers. 

Schneiderman, B. (1998).  Designing the User interface: Strategies for Effective Human-computer Interaction, 3^rd ed.  Reading, MA: Addison-Wesley Publishing.

Smith, P.A., Newman, I.A. & Parks, L.M. (1997).  "Virtual hierarchies and virtual networks: Some lessons from hypermedia usability research applied to the World Wide Web.  Journal of Human-Computer Studies, 47, 67-95.

Spiro, R.J., Coulson, R.L., Feltovich. P.J., & Anderson, D. (1988). "Cognitive flexibility theory: Advanced knowledge acquisition in ill-structured domains," Proceedings of the 10^th Annual Conference of the Cognitive Science Society.  Hillsdale, NJ: Erlbaum.

Wadsworth, B.J. (1972).  Piaget's Theory of Cognitive Development.  New York: David McKay Company, Inc.

Wiersma, W. (1995).  Research Methods in Education: An Introduction. 6^th ed. Boston: Allyn and Bacon.

Wilson, W., Abosabaah, F., Alyoshin, A., Brand, G., Creighton, K., Md Dom, Z.B., Kostik, G., LaPointe, P., & Murray, J. (1995).   Multimedia: Providing Future Leaders with an Innovative Method of Instruction. (unpublished research project), Maxwell AFB, AL: Air Command and Staff College.

Zais, R.S. (1976).  Curriculum: Principles and Foundations.  New York: Thomas Crowell Company.

About the Author:

Dr. Don MacCuish is the Associate Dean of Distance Learning at the Air Command and Staff College, Maxwell AFB, Alabama.  Dr. MacCuish has extensive experience in the design, development and implementation of interactive training and education programs.  He has published several dozen articles on training, curriculum development, and educational and psychological assessment. He can be contacted at ACSC/DLV Phone: 334-953-4936, Fax: 334-953-4003, email: donald.maccuish@maxwell.af.mil