Teaching Introductory International Relations in an Entirely Web-based Environment:

Comparing Student Performance Across and Within Groups

Susan M. Johnson

Abstract:

Utilizing observations on distance education research found in the 1999 report by the Institute for Higher Education Policy "What's the Difference: A Review of Contemporary Research on the Effectiveness of Distance Learning in Higher Education," this case study compares completion rate and exam performance of a web-based section of Global Perspectives-Political Science with a traditional section of the same course taught during the same semester concluding that the web course compares favorably with the traditional section in both instances. Further, the study finds that, while female and male students shared similar characteristics upon enrolling in the web-based section, the female students performed better than their male counterparts.

Comparing Student Performance Across and Within Groups

College student use of the Internet as a research and learning tool has become as ubiquitous today as hours in the library stacks were in the past. Increasingly, students are doing much of their academic research on-line and are receiving supplemental learning materials from their instructors using various e-learning platforms.

Taking advantage of this Internet revolution, colleges and universities are expanding their distance education programs by encouraging academic departments and faculty to offer completely web-based courses. According to the National Education Association's "Survey of Traditional and Distance Learning Higher Education Members" published in June 2000, close to half of all distance education courses are web-based courses. Further, the National Center for Education Statistics disclosed in its 1999 Distance Education Statistical Analysis that 82 percent of higher education institutions plan to begin or increase their use of completely web delivered courses (Lewis, et al. 1999). Clearly, the move towards expanded distance education programs with heavy emphasis on web-based classes is becoming near universal with little likelihood of reversal. This being the case, it is necessary to continue to measure the effectiveness of this education medium in different environments.

The 1999 report by the Institute for Higher Education Policy "What's the Difference: A Review of Contemporary Research on the Effectiveness of Distance Learning in Higher Education" assessed the state of research in distance education and concluded that there are a number of areas where such research is lacking (Phipps and Merisotis 1999). This study, taking into account these conclusions, attempts to address some of these issues in the context of an introductory, interdisciplinary international experience course at the University of Wisconsin-Whitewater titled Global Perspectives.

Specifically, the Institute's Report identified several gaps in the distance education research. Further, the Report stated that a key shortcoming of the distance education research is that it does not take into account extraneous variables when comparing groups of learners. This case study addresses two of the gaps enumerated in the Report:

(1)  differences among students enrolled in distance education classes, specifically, for this study, a web-based class; and

(2)  non-completion rates for students in distance education classes, again, specifically for this study, a web-based class, as well as the use of similar groups when comparing distance education to traditional in-class education.

This is a two-part study. Part One compared students enrolled in a traditional section of Global Perspectives with students enrolled in an entirely web-based section delivered via Web Course in a Box, an e-learning platform that allows for total course delivery through its site, based on common examination questions and course completion rates. Part Two assessed performance of students within the web-based section looking at how gender impacts overall performance. The Institute for Higher Education Policy's 1999 Report states that one of the major shortcomings of distance education research is that it fails to take into account differences among students enrolled in distance education classes. This study will attempt to address that concern.

The University of Wisconsin-Whitewater is a regional, comprehensive University in southeast Wisconsin serving approximately 10,000 students. Global Perspectives is part of the University of Wisconsin-Whitewater's General Education Core Curriculum. It is a required course taken by most students in their first or second year of college. The course profiled here is Global Perspectives - Political Science which operates as a freshman-level International Relations course incorporating geography and international political economy. Beginning in Spring 2000, students were offered the option of enrolling in an entirely web-based section of Global Perspectives. No restrictions, aside from enrollment caps, were placed on students in selecting the web-based vs. the traditional sections.

When asked why they enrolled in the web-based section of Global Perspectives, the most prominent reasons were (1) to try something new, (2) so that they could do the work at their leisure and not have to physically go to class, and (3) to gain experience with new technology. Contrary to what is often the case, students enrolled in the web-based section tended to be fulltime, on-campus students at UW-W. In fact, the mean age of the students in the web-based section was lower than that of the traditional class, 19.5 years and 20.4 years, respectively.

Part 1.

Comparing a web-based and traditional section of Global Perspectives

Part One of this study compares students in the web-based section of Global Perspectives with students enrolled in a traditional, in-class section during the same semester. The two points of comparison are completion rate and performance on identical examination questions. Final grades are not comparable because student assignments were not entirely consistent between the two sections and work was weighted differently.

Prior to comparing these two groups, it is necessary to establish consistency among the groups. In other words, are both groups similar in all ways except that one group studied Global Perspectives in a traditional classroom and the other in a virtual classroom? This was established by comparing mean age, percent of first and second-year students, and overall mean grade point average. For each of the three measures, both sections had similar characteristics. There were no significant differences between the web class and the traditional class on any of these measures.

In regards to age, the mean age for both sections was within one year, with a web class mean age of 19.5 years and a traditional class mean age of 20.4 years¹. In both sections, the overwhelming majority of students were first and second-year students. Ninety-seven percent of the web class and 98 percent of the traditional section were students in their first or second year of school. Further, the overall mean grade point average for each section was nearly identical with the web section overall mean grade point average at 2.69 and the traditional section overall mean grade point average at 2.64. The two sections then, are sufficiently similar to compare.

Methods

As was stated above, the web-based and the traditional classes are being compared in two ways. First, the completion rate for each section will be compared and second, performance on identical examination questions. In both instances, analysis of variance (ANOVA) will be utilized to test for statistical significance.

The comparison of completion rates measures whether students in the web-based section were less' likely to complete the course than were students in the traditional section. Numerous studies have shown that students in distance education classes have higher non-completion rates than do students in traditional classes (Jewett 1997; Cheng, et al. 1991; Powell, et al. 1990).

If students in the web-based section have significantly higher non-completion rates than do students in traditional sections, there are implications for both students and the institution. Since Global Perspectives is a part of the University's Core Curriculum, students not completing the web section will be required to re-enroll in the course in a subsequent semester that may result in an additional financial burden, as well as a longer period to graduation. Additionally, the institution will have to make available a seat to the student repeating the course by increasing the number of sections offered or raising the enrollment caps on existing sections.

Completion was defined as completing the required work in the course and receiving a final grade. Non-completion was defined as a student dropping or abandoning the course after the initial five-day add/drop period at the beginning of the semester². Subsequent to the initial add/drop period, students have five weeks to drop courses. Any student that dropped within that five-week period, requested a late drop³, or abandoned the course and received a failing grade as a result were classified in the non-completion category.

In addition to completion rates, the students in the traditional and web-based sections were compared based on multiple choice exam questions asked of both groups. Three exams were administered to both sections. In regards to administration of the exams, students in the web class had a window of twelve hours in which to complete the exam using Web Course in a Box, however once a student accessed the exam, a 15 minute time limit was activated. Students in the traditional section completed the exam during the regular class period.

For each exam, several identical multiple-choice questions were asked⁴. The questions were a combination of definitions, theory application, and example identification. The questions used in the comparison were drawn from two sources: the textbook testbank and the instructor's personal bank of exam questions. In both instances, the questions covered material that students were assigned to read over the course of the semester. Students in the traditional section received instruction on the tested material in the classroom, while the web-based section was provided with focus questions to alert the students to the points in the chapter that the instructor deemed important, and therefore, likely to be on the exam. Because both sections have similar student characteristics, it is expected that the percent of students that answer each question correctly would be consistent for both groups.

Results

In regards to completion rates, there was no statistically significant difference between the web-based and traditional section (Table 1). Ninety-four percent of students, forty six of forty nine, enrolled in the traditional section of Global Perspectives after the initial add/drop period completed the course, while 90 percent of the students, thirty six of forty, in the web-based class finished the course. This finding is not consistent with the majority of distance education literature, which finds that students enrolled in distance education classes have higher non-completion rates than do students enrolled in traditional classes.

A number of factors may have contributed to the 90 percent completion rate. First, Global Perspectives is a required course, so a student that did not complete the course would have to retake it during a subsequent semester. Second, it is an introductory course, so the material presented should be easily grasped by students regardless of the delivery media. Third, the UW-W drop policy, that does not allow for late drops after the fifth week unless there are extraordinary circumstances, discourages non-completion. Overall, the fact that the completion rates between the traditional and web sections were comparable and the mean student final grade in the web section was 2.9/4.0, with only eight percent earning a D or F, and the mean student final grade in the traditional section was 2.8/4.0⁵, suggests that the overwhelming majority of students were able to successfully complete the work and were not merely victims of university policy or were "sticking the course out" in order to fulfill the requirement.

Aside from a comparison of completion rates, student performance on identical exam questions was also compared. On each of the three exams, the traditional class scored a higher mean percent on the common questions (Table 1). Interestingly, the widest divergence was on the first exam where the traditional class mean was 86.8 percent and the web-class mean was 75.2 percent, a statistically significant difference of 10.6 percent. The differences decreased substantially and became statistically insignificant on exams two and three, with the traditional section scoring 3.5 and 2.2 percent better, respectively. No students dropped the class between exams one and two, so the improvement in web-class performance was not attributable to poorer scoring students dropping out of the class. Further, on exams two and three, the web class scored better than the traditional class on close to 40 percent of the questions. On the other hand the traditional class scored better than the web class on all questions but one on the first examination and for the one questions where the web class scored higher the difference was only two percent.

Given that the two groups of students were quite similar and scored closely to each other on exams two and three, the fact that the web-class scored 10.6 percent lower on exam one may suggest that the medium used to administer the exam impacted student performance on the first exam and performance improved as students became more comfortable with the process. The parameters for completing the exam on-line were quite restrictive. Students received a lengthy set of instructions and warnings meant to discourage cheating and minimize computer/browser/program-related problems that could have arisen during the completion of the exam. Since none of the students had previously taken an entirely web-based course and did not have familiarity taking exams on-line, the experience may have created a certain level of anxiety in students that did not allow them to perform as well on exam one as they did on subsequent exams. Had the divergence been consistent over the course of the three exams, there would have been cause for concern. As it stands, this finding suggests that, just as instructors in traditional classrooms attempt to minimize in-class test anxiety, it is necessary for those teaching web-based classes to be cognizant of student trepidations as they relate to the use of a new technology for assessment purposes.

Table 1. Comparing Completion Rates and Examination Performance
	Web Class	Traditional Class
Completion Rate	90.0%	93.9%
Examination Performance
Exam 1**	75.2%	86.8%
Exam 2	78.8%	81.9%
Exam 3	84.1%	86.3%
N=82
**p<10

Discussion

Overall, when comparing the web-based class and the traditional class on the measures of completion rate and performance on common exam questions, the students in the web class compared favorably with the students in the traditional section. There was no significant difference in regards to completion rate and for the most part, the students in the web class performed comparably to the students in the traditional class on examination questions.

Part 2.

Performance Differences within the Web-based Section of Global Perspectives

In addition to comparing the web-based section to a traditional section of Global Perspectives, it is telling to compare sub-groups within the web-based class to determine what differences exist. Due to the fact that the web class was comprised of primarily first and second-year students of similar ages, comparisons based on grade level and/or age were not possible. One area, however, identified in the literature as one that produces differences in student performance is gender. Powell, et al. (1990) found that females participating in distance learning performed better than their male counterparts. This being the case, the focus on the within group comparison will be differences in the performance of men and women. It is of particular interest to compare the performance of the female and male students in regards to the class assignments that were discussion-oriented.

Historically, a concern for educators has been the dominance of more vocal male students in the classroom at the expense of female students. In an asynchronous web-based environment, the opportunity to speak is not as tempered by overzealous peers, so a student who normally may have been reticent to speak up in a normal classroom setting (females more so than males), may feel freer to express an opinion.

In order to compare the performance of men and women in the class, it is necessary to establish that the groups are, in fact, comparable. That is, are the groups sufficiently similar at the outset so that the researcher can assume that any distinctions that may emerge are not due to preexisting differences? The comparability of the men and women in the web-based class will be measured by looking at the mean grade point average of each group, the mean age of each group, as well as the percent of each group that are first or second year students and self-described traditional students.

In regards to grade point average, the female students had an overall mean GPA of 2.7, while the male students had an overall mean GPA of 2.6. Further, all of the male students were first or second year students and 96 percent of the females were first or second year students. Additionally, the mean age of each group was 19 years old and a majority of the females and males identified themselves in a biographical assignment as traditional, on-campus students. On these measures, there were no statistically significant differences. This being the case, it is possible to compare the performance of the women and men in the web-based section of Global Perspectives.

Methods

The female and male students will be compared in regards to overall class performance a~ well as performance on a number of different types of assignments. Based on the findings of Powell, et al. (1990) it is hypothesized that the female students will earn higher grades than their male counterparts. If this is the case, particularly in the discussion-oriented assignments, it may be argued that computer-mediated instruction allows women to make their voices heard more easily than in a traditional classroom. The mean GPA for each group will be tested to determine if there is a statistically significant difference using a t-test. In addition to final grades, the groups will be compared based on discussion assignments (current events and debates), written assignments (weekly assignments and a major essay assignment) and examinations (subjective and objective questions).

Results

In each of the areas assessed, the female students earned higher grades than their male counterparts. The mean class GPA for the female students was 3.0, while the mean class GPA for the male students was 2.6 (Table 2). The difference in the female and male mean class GPA was statistically significant. The men, then, matched their overall GPA upon entering the class (2.6) while the female students exceeded the overall GPA of their group (2.7). It could be argued then, that the males performed as one would expect, while the females exceeded the expectations one would have set solely based on GPA and performed significantly better than than male students. In regards to the distribution of final grades (Table 2), 38 percent of the female student earned A's, while 20 percent of the male students earned A's. The number of B's for each group was, however, more similar with 38 percent of females and 40 percent of males earning B's. Further, the males earned twice as many C's and failing grades.

Table 2. Comparing Female and Male Final Grades in Web-based Section of Global Perspectives
Overall Performance (GPA)**	Female	Male
Final Grade Distribution	3.0 / 4.0	2.6 / 4.0
A	38%	20%
B	38%	40%
C	16%	30%
D	4%	-
F	4%	10%
N=36
**p<10

As was stated above, the groups were compared not only on overall performance, but also on grades earned completing a number of assignments. In each instance, the female students earned higher mean grades (Table 3). In the discussion assignments, the female students scored, on average, seven percentage points better than the males. These discussion assignments involved posting and discussing current events and debating current issues in international politics after completion of the assigned readings. Further, the women performed better on the written assignments as well, particularly on the weekly 2-3 page analytical essay assignment. Finally, in regards to exams, there was less than a two percentage-point difference between the men and women's performance on the three examinations.

Table 3. Comparing Female and Male Final Grades on Assignments and Examinations in the Web-based Section of Global Perspectives
Assignments	Female	Male
Discussion Assignments
Current Events	84.9%	77.8%
Debates	85.5%	77.8%
Written Assignments
Weekly Assignments	84.7%	77.7%
Major Writing Assignment	85.5%	82.3%
Examinations	82.1%	80.6%
N=36

Discussion

These findings illustrate that the female students performed better than the male students in all instances and by a half letter grade or more in most cases. The male and female students entered the class with similar GPA's, yet the female students earned a mean GPA in the web class of 3.0 while the male students earned a combined 2.6. As was stated above, this was a statistically significant difference in the performance of the two groups. In regards to assignments, the largest differences were found in the discussion-oriented and critical thinking assignments. The fact that the female students performed up to a half letter grade better than their male counterparts supports the findings of Powell, et al. (1990) that female students tend to earn higher grades than males in distance education courses. It is difficult to draw conclusions and generalize from this case study alone given the small sample size, which does not lend itself to significance testing. However, these results and those of others who have found that women perform better than men, do suggest that more research should be done in regards to gender differences in attitudes towards distance education and performance in distance education courses.

General Conclusion

This study, analyzing a web-based section of an introductory, international experience course, and how it compared to a traditional section of the course, as well as differences in the performance of women and men in the web-based section,

(1)  addressed some of the concerns identified in the Institute for Higher Education Policy "What's the Difference: A Review of Contemporary Research on the Effectiveness of Distance Learning in Higher Education" Report,

(2)  supported the findings of many previous studies that indicated that distance education courses match up favorably with traditionally delivered courses, and

(3)  identified areas where more research is need a In the aforementioned Report by the Institute for Higher Education Policy, the authors stated that the literature assessing and describing distance education is substantial. However, due to the fact that more and more colleges and universities are moving full-speed ahead with web-based instruction, this is an area of distance education where more research is necessary in order to ensure that this medium of course delivery benefits students and identifies those students that may have a greater likelihood of success in web-delivered courses so that programs are implemented/modified with this information at hand.

Endnotes

¹ One outlier in the traditional section was responsible for a large part of the difference that did exist. Excluding that student from the calculation of mean age leaves the traditional section mean age at 20.0 years.

² During the five-day add/drop period at the beginning of the semester, students add and drop courses for a variety of reasons often not related to the course material or delivery of that material. This being the case, the researcher decided that a student who drops the course during that period would not be classified as a student who did not complete the course. Instead, students whose names appeared on the updated official roster subsequent to the add/drop period were used to measure completion rates.

³ A late drop is permitted only when a student has a significant and verifiable reason for being unable to complete the course (e.g. serious illness of student or close family member, emotional problems, etc.). Poor performance is not an accepted reason for granting a late drop. The decision to grant a late drop rests with the dean's office.

⁴ The number of identical questions varied for the three exams based on the fact that the web-based class completed shorter multiple-choice components than did the traditional class and the traditional class was tested on additional materials, outside readings and current events, while the web-based class was not. On exam one, 13 questions were identical. For exam two, 11 questions were identical. Exam three included eight identical questions.

⁵ As was stated earlier, the final grades for the two sections are not directly comparable because there were variations in assignments and weighting. However, the general expectations were the same and one course was not more difficult than the other, just different in some ways. Direct comparison for the point of drawing conclusions about the performance of each class, then, is not appropriate. However, in order to establish that the web class students generally did no better or worse than the traditional class students, the average final grade for each is provided.

References

Barry, M. and G. Runyan. 1995. A review of distance-learning studies in the U.S. Military. The American Journal of Distance Education (9)3: 3747.

Cheng, H., J. Lehman, and P. Armstrong. 199 1. Comparison of performance and attitude in traditional and computer conference classes. The American Journal of Distance Education 5 (3): 51-64.

Jewett, F. 1997. The human computer certificate program at Rensselaer Polytechnic Institute: A case study in the benefits and costs of a joint industry/university designed program featuring integrated delivery systems. Seal Beach, Ca: The Chancellor's Office, California State University.

Lewis, L., K. Snow, E. Ferris, D. Levin, and B. Greene (project officer). 1999. Distance education at post-secondary institutions: 1997-98. NCES 2000-013.

Martin, E. and L. Rainey. 1993. Student achievement and attitude in a satellite-delivered high school science course. The American Journal of Distance Education (7)1: 54-6 1.

National Education Association. 2000. A survey of traditional and distance learning higher education members. Available online at http://www.nea.org .

Phelps, R., R. Wells, R. Ashworth, Jr. and H.. Hahn. 199 1. Effectiveness and costs of distance education using computer-mediated communication, The American Journal of Distance Education 5(3): 7-19.

Phipps, R. and J. Merisotis. 2000. Quality on the line: Benchmarks for success in internet-based distance education. The Institute for Higher Education Policy. Available online at http://www.ihep.com.

Phipps, R. and J. Merisotis. 1999. What's the difference? A review of contemporary research on the effectiveness of distance learning in higher education. The Institute for Higher Education Policy. Available online at http://www.ihep.com.

Powell, R., C. Conway, and L. Ross. 1990. Effects of student predisposing characteristics on student success. Journal of Distance Education 5(l): 20-37.

Russel, T. 1999. The no significant difference phenomenon. Chapel Hill, NC: Office of Instructional Telecommunications, North Carolina State University.

Souder, W. 1993. The effectiveness of traditional versus satellite delivery in three management of technology Master's degree programs. The American Journal of Distance Education 7(l):37-53.

About the Author:

Dr. Susan Johnson is an Assistant Professor in the Department of Political Science and Coordinator of the Public Policy and Administration Program at the University of Wisconsin-Whitewater in Whitewater, Wisconsin 53190

Phone: 262-472-4766, Fax: 262-472-2803 (fax for entire floor, please notify if materials are faxed) email: johnsons@mail.uww.edu