'A' is for Assessment:

Identifying Online Assessment Practices and Perceptions

Mary I. Dereshiwsky

Introduction

Assessing student performance is an admittedly challenging aspect of instruction. Often equal parts art and science, it can cause anxiety for students and instructors alike. Are the assignments a valid reflection of the course curriculum? Is there an equitable and clearly understood evaluation and feedback system in place? Above all, over and beyond a formal grade…is the assessment genuinely meaningful and useful to students in terms of their academic growth?

The online teaching/learning environment poses some special challenges with regard to assessment of student performance. Teachers lack the informal, yet often helpful, visual and verbal clues as to how their students are processing the lessons. Furthermore, some traditional methods of assessment, such as the timed in-person closed-book classic examination, are no longer feasible in their present form in the Web-based classroom.

Literature in the area of assessment of online learning has identified the desirability of multiple alternative forms of such assessment. Paloff and Pratt (1999) advocate the use of both quantitative and qualitative forms of evaluation of online students’ work. They also recommend that students have input into their own evaluation process, including negotiated identification of differential student contributions in the case of group assignments. According to Schwartz and White (2000), feedback to Web students should focus on specific behaviors rather than individual personalities; be oriented towards the informational needs of students; be directed towards changeable behavior; be empowering to students such that they feel they truly own the feedback; be clear, accurate and timely; and constitute part of an ongoing relationship between instructor and student. If this sounds identical to desired characteristics of all assessment—including that of the traditional face-to-face classroom—this would not be surprising to some researchers. Zvacek (1999) asserts that “good practice” in assessing attainment of student learning may indeed be constant across different classroom and delivery modalities. Nonetheless, she acknowledges the relative efficiency of feedback to students that is possible in alternative technological delivery methods. Jarmon (1999) states that online instructors report a greater perceived sense of their individual students than in the traditional group classroom setting.

The present study seeks to identify key issues pertinent to assessment of students’ performance in Web-based and Web-enhanced coursework. It constitutes a qualitative case study focused on the Educational Leadership department at the Center for Excellence in Education at Northern Arizona University (NAU) in Flagstaff, Arizona. Founded in 1899 as the Northern Arizona Normal School, NAU today serves the needs of nearly 20,000 students. It offers 100 bachelor’s, 40 master’s and 8 doctoral degrees. More on the history of NAU can be found at www.nau.edu/history.html. The geographically dispersed student population throughout Arizona is served via NAU’s distance education programs. These include interactive television (IITV) and satellite course offerings. Web-based and Web-enhanced undergraduate and graduate courses have been formally offered since the Fall of 1997 via NAUOnline (www.nau.edu/nauonline). At the present time 200 courses are being offered to 10,000 students via IITV and the Web.

Methods and Procedures

A survey was disseminated via e-mail to the five Educational Leadership faculty who are teaching courses on the Web via NAUOnline. This survey contained open-ended questions asking the faculty member to identify what courses he/she teaches online; what assessment procedures he/she uses; how these compare to student assessment in equivalent face-to-face classroom-based instruction; perceived strengths and weaknesses of current online assessment procedures; and the faculty member’s wish list regarding online assessment.  All five Educational Leadership faculty teaching online completed and returned the survey via e-mail. As a qualifying Educational Leadership faculty member, the researcher also completed the survey independently and prior to analyzing her colleagues’ responses.

The entire eligible population was canvassed for this study and consisted of 6 total subjects. Brewer and Hunter (1989) suggest that there is an inherent trade-off in quantitative vis-à-vis qualitative data analysis. They indicate that the greater volume and richness of detail of qualitative responses compensates in many cases for a smaller overall sample size. Furthermore, the descriptive case study nature of this study implies acquiring baseline information on ‘what is’ regarding issues in online assessment. According to Yin (1994), this represents a promising start in researchers’ understanding of a given contextual phenomenon: one which may be further verified via cross-case analysis and replication of the individual case study results at other times and in other settings.

The results of the qualitative assessment survey will be summarized below according to key themes of inquiry. They will also be illustrated in summary tabular format.

Results

Courses Taught Online

Web-based courses taught by the Educational Leadership faculty represent a cross-section of master’s and doctoral level program offerings. These include leadership skills; leadership development; history of American education; publicity and politics in education; critical issues in education; administrative internship; the community college; introduction to research; research design; qualitative research; dissertation seminar; and doctoral dissertation advisement of individual students. These courses represent a mix of requirements and electives in educational leadership graduate study. They also comprise a mix of levels, from mastery level material to higher-order independent learning practicums.

Assessment Methods Used Online

A variety of procedures are employed by Educational Leadership faculty to assess students’ learning in Web-based classrooms. These include the following:

The preceding represent a wide cross-section of type and level of assignments that seem at first glance virtually indistinguishable from the variety of assessments utilized in the traditional face-to-face classroom setting.

Comparison of Web Student Assessment to Traditional Student Assessment

Related to the above observation, two survey subjects stated that they do not perceive any significant differences between online assessment and that of the same courses taught in the traditional face-to-face setting. As one respondent put it, the assessment methods are the same; it’s just the experience of the students that differs due to the type of classroom setting (traditional vs. Web-based). Another faculty member observed that the content of learning is identical, but the way that students process the curriculum differs. Web students learn by reading, visiting other related Websites for supplementary information and responding to short-answer questions, according to one faculty member.

The faculty identified a number of specific examples of such comparability. Group interaction is built in via the Virtual Conference Center, simulating that of the live classroom. Likewise, having a set of fixed due dates staggered throughout the semester effectively simulated the continual and periodic student engagement required in the traditional classroom format. One faculty respondent also pointed out that, regardless of classroom setting, the key criteria for effectiveness are sensitivity, caring and insight into individual student differences on the part of the instructor. Likewise, this faculty member noted that “both motivated and manipulative students” can succeed in the Web environment just as in the face-to-face classroom.

The faculty surveyed also attributed a number of perceived advantages to the Web environment. These included greater efficiency and more timely return of student assignments, and the absence of fixed time limits such as in-class tests on the Web, which in turn encouraged greater student reflection in learning and care in preparation of assignments. The absence of timed tests and greater flexibility in pacing one’s learning and response to assignments also significantly reduced student stress, in the words of one survey respondent. Another faculty member observed that the Web materials eliminate the cost involved in duplicating student handouts to be distributed manually in the live classroom. 

At the same time, the surveyed faculty identified a number of differences in assessment between traditional face-to-face and Web-based instruction. These included a greater average number of individual student assignments on the Web (15 to 25, as compared with 3 or 4 in a live classroom, according to one faculty member); some difficulties in coordinating student interaction for group work on the Web; and the fact that all students in a Web-based classroom are required to participate in some way, as compared with a student in a traditional class who may choose to be more passive and not interact.

One advantage of the live classroom, according to one faculty member who has taught in both formats, is that it is easier to assess students’ understanding and mastery of current learning material. He/she can do this more conveniently in the face-to-face group setting via such means as oral summaries, impromptu mastery learning quizzes and quick assignments.

 

What  Has Worked Well in Web Course Assessment

Efficiency in assignment review and return was a key emergent theme regarding the advantages of assessing student progress on the Web. One faculty member noted that his/her course can now be successfully taught in 2 1/2 weeks due to the convenience and speed of information exchange in the Virtual Conference Center. Web course students appreciate the speed of assignment return coupled with the detailed annotated comments, thereby providing them with greater individual attention than is sometimes possible in the live group classroom. Students also like the convenience of the click-and-send format of submitting their assignments. The Virtual Conference Center was also recognized as an effective interactive medium for simulating group exchange.

Another relative plus of the Web classroom is the additional time that students have to process the learning material and respond to the assignments. They can self-pace a bit more conveniently than in the traditional classroom with its timed tests, and also thereby experience less stress and time pressures.

These advantages were coupled with affective benefits of Web teaching and learning. Surveyed faculty lauded the ability of the Web classroom to facilitate candor and trust in student-instructor communication, as well as greater risk taking by students in acquiring new learning skills such as technological navigation.

 

What Has Not Worked Well in Web Course Assessment

The surveyed faculty identified a number of persistent difficulties in response to this question. They center on student/faculty readiness for online learning, misconceptions regarding the nature of the Web classroom and how it may differ in format from traditional instruction, related lack of needed administrative support, and some technological glitches in communication.

Some students tend to inappropriately self-select into online classes, according to survey respondents. They may mistakenly believe that Web courses are easier due to the lack of periodic face-to-face required group meetings. It is important for students to take the time to carefully assess their individual respective learning styles in order to determine if the Web classroom is indeed the right or best fit for their needs. One instructor also noted the great diversity in student backgrounds and readiness levels made it difficult to facilitate meaningful group interaction online. Another survey respondent has found it difficult to assign papers in the online classroom.

In addition, both students and faculty have experienced persistent startup problems with the technology. One faculty respondent bemoaned the fact that he/she “…stumbled through…” his/her first two courses without adequate training in online navigation and use of the VCC. Related student deficiencies, as noted by online instructors, appear in the form of in adequate writing skills. These include principles of correct grammar and knowledge of proper formatting style for manuscripts such as final papers.

Common technological problems include reliably keeping track of a greater diversity of individual and group online student assignments. This has been coupled in some cases with student impatience in expecting immediate feedback to submitted work on the Web. On a related note, another faculty member bemoaned a typical “e-mail avalanche” from students in inquiring about whether assignments had been received or graded yet. Students also tended to repeatedly e-mail their instructor regarding questions that had already been answered in the VCC in the “Announcements and Updates” or “Questions and Answers” folder. This e-mail backlog made it difficult for this particular instructor to reach and return submitted assignments in the desired 24-hour turnaround time period.

A final category of problems had to do with administrative misconceptions and related lack of needed support for online teaching. These included too-large minimum class sizes (with course caps twice the size of recommended class maximum for Web classroom), as well as repeated reductions in grader funding support. These also greatly hindered the faculty member’s ability to respond quickly and in desired detail to individual student submissions on the Web.

'Wish Lists' with Regard to Online Assessment

The Educational Leadership faculty responded to this survey question with a broad range of desired improvements. These can be roughly grouped into technological access and preparatory issues, overall assessment needs that transcend individual learning environments, and increased administrative and peer support.

With regard to technological issues, the surveyed faculty members wished for greater improved and reliable Internet access to remote geographical areas where many students reside. They also proposed improved training for online students and faculty with respect to hardware and software use. One faculty member also suggested creation of an automatic grade book that can be readily accessed by students as well as faculty. He/she observed that this would cut down on the number of student e-mail inquiries regarding their grades to date in the course. Another suggestion dealt with access to “boiler plate responses” to facilitate more efficient grading of mastery-level fixed choice assignments via the copy and paste function.  Another respondent suggested a greater move towards group work to help reduce the current “e-mail glut” that often prevents rapid turnaround of student assignments. He/she expressed a desire to continue to wean students away from indiscriminate use of e-mail for non-emergency, non-personal messages. This faculty member believes students need to be encouraged instead to rely on the VCC Questions and Answers and Announcements and Updates folders, checking this common workspace periodically in the event their individual question has already been answered or to post their question for all to see. Finally, one survey respondent feels students need to be more realistic—and patient—regarding realistic instructor turnaround of assignments and messages. Technology sometimes acquires an aura of instantaneousness that engenders such student expectations.

Not surprisingly, given the preceding discussion on the equivalence of assessment issues across classroom structures, some wish-list items apply equally to the online and face-to-face classroom. Faculty suggested continued emphasis on clearly understood grading standards in general, as well as some individual instructor leeway to depart from announced standards without bias to better meet individual student needs. Students also need to better self-assess both their readiness for learning on the Web as well as their ongoing progress in the course. One survey respondent suggests the desirability of moving to a pass-fail evaluation system in part as a response to the increased prevalence of grade inflation.

Peer support emerged as a factor in the wish-list responses. One faculty member wished for continued mentoring and communication among faculty with regard to teaching on the Web. Another respondent would like to see greater understanding and respect overall for fellow faculty who take on the unique challenges of the cyber-classroom.

Such peer support was coupled by a desire for greater administrative support. Class size caps need to be adjusted in line with current empirical research regarding the optimal class size for Web-based interaction. Assessment procedures would also benefit from increased funding for grader support. Both of these would help ensure timely return of assignments as well as greater perceived attention to individual students’ learning needs.

Finally, one faculty member underscored the comparability of different classroom structures as well as the overall goals of good teaching by saying: “The way we teach, even online courses, is so outmoded in terms of what we know about adult learners and andragogy.”

Table 1 visually depicts the key emergent themes regarding issues in online assessment as perceived by faculty members.

Table 1.

Summary of Issues Regarding Online Student Assessment:

Topic	Summary of Survey Responses
Types of Courses Taught Online	Both master’s and doctoral level courses Both required courses and electives Both mastery-level learning and advanced application and analysis courses
Assessment Methods Used	Traditional examinations by prior arrangement Mastery-level individual and group assignments Higher-order learning activities (i.e., case studies, interviews, discussions, papers, research proposals) Creative learning activities (online crossword puzzles, scavenger hunts, evaluation and development of Web pages)
Comparison of Online and Traditional Classroom Student Assessment	No real difference overall Web assessment allows for more efficient and detailed feedback to students Web learning allows for greater individuality in terms of student self-pacing Web learning results in more individual assessments of student work Web learning encourages greater student participation and engagement Traditional classroom allows for greater instructor leeway in gauging ongoing student understanding of learning material
What Has Worked Well in Web Course Assessment	Efficiency in timely assignment return More detailed individual feedback to students Convenience of communications between instructor and students (“click and send”) VCC simulates effective group exchange among students Additional time for students to process learning and complete assignments Greater opportunity for students to develop technological skills in general
What Has Not Worked Well in Web Course Assessment	Student misperceptions about workload in online classroom as compared to traditional classroom Lack of student self-assessment of learning style and appropriate match to the Web classroom Startup problems with the technology for both students and instructor Student deficiencies in writing skills E-mail glut preventing timely and efficient return of assignments by instructor Too-large mandated minimum class sizes Insufficient grader funding support to help facilitate timely return of assignments and other course feedback
Wish Lists Regarding Online Assessment	Greater technological access and support for instructor and student readiness Improved technological assessment procedures (i.e., online grade book) Increased student reliance on common work areas (VCC folders) to discourage e-mail glut Increased student understanding and patience regarding ‘reasonable’ turnaround time for instructor feedback on assignments Increased administrative support regarding maximum class sizes and grader funding Increased peer mentoring, support and understanding regarding issues faced by online faculty Increased incorporation of current adult learning theory and practice into teaching methods

 

Discussion

Based on the preceding, a number of conclusions can be drawn regarding online assessment procedures:

This initial descriptive qualitative case study should be replicated in other academic departments and settings offering online instruction. Through such cross-case analysis, we can continue to discover which principles of online assessment seem to hold true across diverse settings and which results are more localized in nature to a particular campus or academic department.

As a popular quote puts it: “The only things that count are the things that get counted.” Those of us who are pioneers in technologically mediated instruction owe it to ourselves and our students to continue to explore ‘what counts’ and to make it as compatible with student needs and sound academic practice as possible.

References

Brewer, J. & Hunter, A. Multimethod research: A synthesis of styles. (1989). Thousand Oaks, CA: Sage Publications, Inc.

Jarmon, C. (1999). Testing and assessment at a distance. In Boaz, M., Elliott, B., Foslee, D., Hardy, D., Jarmon, C., & Olcott, D (Eds.), Teaching at a distance: A handbook for instructors. Mission Viejo, CA: League for Innovation in the Community College.

Paloff, R.M. & Pratt, K. (1999). Building learning communities in cyberspace: Effective strategies for the online classroom. San Francisco, CA: Jossey-Bass, Inc.

Schwartz, F. & White, Ken (2000). Making sense of it all. In K.W. White and B.H. Weight (Eds.), The online teaching guide: A handbook of attitudes, strategies and techniques for the virtual classroom. Needham Heights, MA: Allyn and Bacon.

Yin, R.K. (1994).  Case study research: Design and methods. Thousand Oaks, CA: Sage Publications, Inc.

Zvacek, S.M. What’s my grade? Assessing learner progress. (1999, November). Tech Trends, Vol. 43, No. 5.

About the Author

Dr. Mary I. Dereshiwsky is an Associate Professor of Educational Leadership and Research at Northern Arizona University’s Center for Excellence in Education in Flagstaff, Arizona. She can be reached via e-mail at LDRSPETSCherry@aol.com and by telephone at (520) 523-1892.