Editors Note: This paper is an excellent example of how Universities can work with industry to serve their mutual needs.

Distance Learning Software Usefulness and Usability: User-Centered Issues in Practical Deployment

By Cindy Ruman and Jay Gillette [1]

Summary

Blackboard CourseInfo is a software package that seeks to fill a niche in the distance learning market by bringing together various applications.  Applications required include communication systems and a means to store and access documents. The question is how useful and usable is the Blackboard CourseInfo software to the students and faculty.  This paper defines the key elements that are used within an on-campus graduate class and gives the results of surveys of that class on how useful and usable Blackboard CourseInfo was in supplementing the educational experience.

This study examined the usability of Blackboard through three approaches: user group interviews, usability testing sessions, and an end-user survey.  In laboratory usability session, at least eight of nine participants were able to complete all of the tasks, taking, on average one to two attempts.  Participants voiced their concerns and opinions through follow-up questions and survey ratings.

The most common response made by the study participants focused on the categorization of utilities within each menu button.  Many tasks were not in the place that seemed intuitive.  The most popular feature was the grade checking ability.  Mean rating from the survey on ease of use and satisfaction were 3.58 and 3.56 respectively.

According to the survey, the most useful and usable item was the ability to check grades.  Yet, ironically this is one of the most problematic items for professors to implement within their class

Staff and faculty interviews showed a reasonable degree of satisfaction, yet adjustments still need to be made, and the comfort level for this new software deployment will need to be increased to fulfill the needs of the users.

Introduction

The higher educational realm is expanding from the classrooms of colleges and universities into the homes of collegiate students.  Yet this new era of education may place students at a disadvantage because of a lack of face-to-face interaction with the professor. 

Anne B. Keating (1999, pp. ix-x) has a practical approach to working in Internet-aided instruction, in her Wired Professor guide.  She says:

The rational for using the World Wide Web to deliver course material, as well as to promote classroom interaction, rests in the potential that this technology has for providing twenty-four-hour access to information from any computer connected to the Internet. . . . By the time you read these words, the Web and related online technologies will have been further enriched by capabilities that we can only guess at now.  This is the challenge of the Internet—the moment you take it on you become a participant in its development.

Blackboard CourseInfo is a widely deployed software that attempts to fill this potential by creating a package to assist in distance education by providing course material access and communication links.

Blackboard CourseInfo was designed to assist the interaction of students by providing multiple communication possibilities.  Two asynchronous features provided by Blackboard are email and discussion threads.  The use of the discussion threads also helps in the posting of a question or comment and allowing others to respond to the posting.  The progression of the discussion can be followed chronologically.  Synchronous communication is provided through a virtual chat feature.  This is time-constrained but offers instant feedback for the students that are participating in the chat room.

Blackboard offers features to assist faculty that use the software by providing sections to post announcements, assignments, documents, and maintains a grade book.  These sections provide information about any announcements or assignments made in class, eliminating the restrictions of distance that would prevent the students from accessing this information. For example, the Documents section is where professors can post syllabi, lecture notes, and other documents that may be needed by students.  Another feature is the Grade Book section where the professors are able to post students’ grades allowing them to find their current status in class in accordance with the points that they have achieved versus the points possible.  

Blackboard CourseInfo also is focused upon information that the students can provide themselves.  The students have the capability to host a web page about themselves to allow other students within the class to know about their fellow classmates.  This simulates a community that can be found in the face-to-face classroom.  A further feature that is controlled by the students is the Group section.  This allows a group of students to collaborate together and assist each other in assignments or projects, simulating other interaction that may be found in a normal classroom situation. 

Research Methods

Participants

Study participants included faculty, technical support staff, and graduate students in the various phases of the study.  We expected that all participants had some knowledge of the product from periodic use during  (at least) the past three months.  The following summarizes the numbers of participants in the parts of the study.

Interview - 6 participants

Usability Testing Sessions - 9 participants

Survey - 30 participants

For the interview phase, participants were chosen from three user group levels: student, faculty, support staff.  Different levels of users give a broader perspective of impact on users.  The chart below gives some characteristics of this group.

	Gender	Age Range	Education	Role	Product Experience
1	M	50 - above	Ph.D.	Faculty	2nd semester of use
2	M	50 - above	Ph.D.	Faculty	1st  semester of use
3	F	35-50	M.S.	Support	2nd semester of use
4	M	20-35	M.S.	Support	2nd semester of use
5	F	20-35	B.S.	Student	1st semester of use
6	M	20-35	B.S.	Student	2nd semester of use

For the usability testing sessions, volunteers were sought from a graduate course that utilized Blackboard during the present term (MGT 500 - the instructor of this course allowed our group to contact students).  Nine volunteers participated in the sessions.  Research shows that small numbers of users are sufficient to uncover most usability problems and that subsequent test users find the same problems already discovered (Nielsen, 2000).

A survey was given in the above course during one of the class sessions.  The following are demographics about these participants.

Number of respondents	30
Mean age (SD)	28 (5.6)
Status	24 full-time, 6 part-time
Sex	20 males, 10 females
Previous Blackboard use?	16 yes, 14 no
Average use per week (SD)	1.25 (.8)

Context of Product Use in the Test

Tasks

The study included interviews conducted in person or by phone to ask about the experiences the different user groups had with the Blackboard software. 

During the usability testing sessions, evaluators observed users at the computer performing pre-defined tasks and then asked follow-up questions.  The tasks selected were representative of the types of features common to student use or potential use.  Participants were given a written task list/instruction sheet for use during the session (see Appendix D).

A survey was given to assess subject ratings of product features.

Test Facility

Interviews were conducted in the participants’ offices or in another classroom environment.  The usability testing sessions were held in a computer lab.  No video or audio recording was conducted.  The survey was administered in the regular classroom.

Participants’ Computing Environment

Participants in the usability testing sessions accessed Blackboard CourseInfo from any web browser (typically Internet Explorer or Netscape).  Microsoft Word and PowerPoint software must also be loaded on the computer if such software was used by the professor to create materials for Blackboard.  In the absence of these programs, participants would need knowledge of conversion techniques for other programs.  Participants in this study used PCs (Apple computers also work) with monitors, and used the keyboard and mouse for input devices.

Test Administrator Tools

A customized questionnaire was compiled and given to those present in the graduate class (MGT 500).  This survey can be found in Appendix A.  Usability testing sessions were hand recorded on observation sheets (see Appendix B).

Experimental Design

Procedure

An instruction sheet for the evaluators was prepared so that all evaluators would be consistent in the way they conducted the sessions (see Appendix C).  Participants were tested four in a group in a computer lab with one evaluator available to observe each subject.  No interruptions were encountered in the use of this lab.  Participants were greeted, the session purpose was explained, and their voluntary participation was acknowledged which included the right to leave the session at any time.

The evaluators then explained that subject would be given a list of tasks to complete, that the evaluator would be making notes of the session but was not judging the person.  It was explained that the evaluators would not answer questions or assist in task completion but would intervene if there were a computer problem that prevented subjects from continuing.  Participants were encouraged to “think aloud” and were given the opportunity to ask questions before the session began.

Evaluators marked task completion, number of attempts made to complete each task, and verbal comments made during the session.  When participants concluded, follow up questions were asked and the participants thanked for their time.

Participant Instructions

The instruction sheet given to participants detailed login procedures, reminded them to think aloud, and enumerated each task they were asked to perform.  Final items included logging out, answering questions, and a thank you for their participation (see Appendix D).

Usability Metrics

The evaluation team endeavored to assess the usability, usefulness, accessibility, and satisfaction encountered by participants using Blackboard CourseInfo.

Effectiveness

This metric assesses the extent to which users can interact with the interface to complete a task.  We marked whether or not tasks were completed to obtain a percentage of achievement.  Similarly, the number of attempts was tallied to give an indication of the effort involved in understanding the operation of the interface to accomplish a goal.  Along with these tallies, participants’ “think aloud” comments were recorded to gain insight into their thought processes as they worked through tasks.

Satisfaction

The extent to which a user is satisfied with a product could impact future use and productivity.  Subjective responses were obtained in all phases of the study.  These include comments during the interviews and from the usability testing sessions, and ratings on the survey administered to the graduate class.  The dimensions of satisfaction, usefulness, and ease of use were included in the survey, both in general questions and those relating to particular features.  The survey utilized a 1-5 Likert scale with 5 meaning high or positive.

 

Results

Interviews

Student Interviews

Student A found the menu items on the navigation bar to be vague.  She expressed the desire to have a “pop-out” menu appear when the mouse points to each item.  She also felt a more developed group area would be helpful.  She did find the email and discussion boards to be beneficial features.  She felt the weekly announcements could be very beneficial if the professor utilized them, or if they could link to the syllabus and automatically list that week’s readings and topics.  Although she does not view Blackboard as a truly necessary tool to be successful in this class, she finds it helpful to have access to the Power Point slides.

One student in our interview group, Student B, mentioned he heard about Blackboard for first time in ICS 602:Human Communication (where it was available as an option), but never used it. He used it first time in this semester in MGT 500. He usually uses Blackboard once to twice a week to download the class handouts, to read assignments and announcements, and to check his grade. He said he has never had any problems logging on the program. It is easy to access. However, one problem he experienced is that there are too many sub-options under each button of the menu selection on the left-hand side of the main page. That is confusing.

Also, the system should show those sub-options on the main screen telling which is under which. So, the users do not have to do multiple steps hoping around between pages. He said, “If you don’t know the system, you have to look through in every button to find what you are looking for.” He suggested to do a better menu selections that shows sub-menus or the menu hierarchy that can be seen on the main page by rolling mouse without going to another page of every option. In overall he said “functionality” is good. The software assists him to complete his jobs. Also, he likes the “weather report” feature showing on the main page. It’s cool!

Support Staff Interviews

Interviews with Staff Member A of University Computing Services, and Staff Member B of the Center for Teaching Technology yielded many things.  First, it afforded a perspective of Blackboard through the eyes of the support staff for Blackboard.

Some of the main problems that Staff Member B heard about from the users was both faculty and students seemed to forget their passwords quite often, and as a result they have to call for technical support. Staff Member B suggested possibly having an online drop box or complaint box.

Faculty Interviews

We interviewed two professors on campus to gain a faculty perspective on the usefulness of CourseInfo, the pitfalls, as well as needed improvements or innovations in the implementation and use of the Blackboard CourseInfo software.  For confidentiality, the professors here will be referred to as Professor A and Professor B. This was Professor A’s second semester using the CourseInfo software, whereas the use of the software was a new experience to Professor B. 

Professor A and B mentioned some key benefits of the use of the software. Both discussed items such as the use of the bulletin board and posting abilities.  They found that discussion could and would occur through this medium as well as it being a good place to disseminate information.  Professor A utilizes the online assignment submission feature heavily as a way to grade and return papers quicker.  Professor B liked the ability to post slides from class lectures as well as the use of the grade book system.  Professor A, however, heard “horror stories” about the Grade Book and prefers to maintain the current system.

Both professors agreed that a considerable amount of time was required to prepare for a class, yet Blackboard focused their planning and organization.  The extra work seemed to be appreciated by the students.  The professors mentioned that their classes enjoyed being able to access the CourseInfo site and gain information about their course.

As with any implementation and use, concerns do arise.  There seemed to be several issues that both professors voiced opinions about.  First, simply managing the courses seemed to be a struggle. The implementation and initial setup seemed to be acceptable or bearable, but the daily maintenance or management of student accounts became troublesome for them.  Secondly, in a scenario given, if a professor wished to send an email to all of the students, concerns arose as to the difficulties and time it took to change each student’s email address if he or she did not use the default Ball State email system.  And for those who did use the Ball State email account, confusion arose between those still using the VAX system or those on the newer Microsoft Outlook system.  This situation gave rise to multiple possibilities and difficulties in assuring that all students had updated and accurate email addresses for the course. 

Third, Professor A mentioned that this was felt especially by non-traditional students who found it difficult to come to campus during office hours to acquire the needed VAX account and then access the appropriate people if questions or difficulties arose.  Fourth, according to the professors, the sharing of written documentation and work should be much easier.  Often, students use different word processing programs, making it difficult to open and read assignments sent to them.  Fifth, a concern exists that there is no virus detection system or spell check for composing email or bulletin messages.  And finally navigation seemed to be a point of difficulty for the professors. It takes many more steps, and waiting between clicks, to do something in Blackboard compared to the old way of doing things.  Professor B mentioned that a nice feature to implement might be to add voice recognition.

This need for help in navigation appeared both in trying to find the correct place to go within the control panel to make changes as well as gaining enough access to support and training from the Center for Teaching Technology.  Each professor gave ideas pointing to having written documentation, a question hotline, a student-focused help area, or somehow implementing a better-structured support and training system.  However, both professors see value in the software, are committed to continue with the use of Blackboard CourseInfo in their future classes, and are optimistic about how the software could facilitate the educational process.

Usability Testing

The following table presents the results of the usability testing sessions.  The collective number of students completing each task, the mean number of attempts to do so, and any problems encounters or comments expressed during the test are shown.

 

The following are comments made in response to the follow-up questions after the testing sessions.

What parts of the interface were confusing, unclear, or hard to use?

What would you like to change?

What did you like about the interface?

Any other questions/comments?

Survey

Means and standard deviations were calculated on all survey questions.  The following table lists the results.  If respondents had not used a feature, they were instructed to leave the question blank.  Therefore, it was not always equal to 30 and is included in the last column.

Question	Mean (SD)	N
How encouraging was your instructor	3.86 (.93)	30
Rate ease of accessing over browser	3.90 (.84)	30
How well does layout allow you to find needed tools	3.40 (1.10)	30
Does the software assist you with tasks	3.00 (1.18)	28
How would you rate the help screens	3.12 (1.40)	16
Perception of the number of required steps	3.42 (.69)	28
How beneficial that software brings together various utilities	3.76 (1.10)	26
How would you rate ease of use of interface	3.58 (.94)	29
Would you like to use in future classes	3.70 (1.20)	30
What is your overall satisfaction	3.56 (1.07)	30

Additionally, we asked for ratings on usefulness and usability of the features in particular.

Feature	Useful Mean (SD)	N	Usable Mean (SD)	N
Virtual Chat	2.70 (1.44)	17	2.86 (1.06)	15
Discussion Threads	3.84 (.94)	25	3.47 (1.03)	23
Personal Web Page	2.84 (1.53)	19	3.23 (1.20)	17
Group Web Page	2.62 (1.50)	16	2.69 (1.10)	13
Checking Grades	4.42 (.98)	26	4.04 (1.08)	24
Submitting Assignments Online	4.07 (1.17)	27	3.62 (1.13)	24
Announcements	4.24 (1.02)	29	4.11 (1.08)	27

Frequencies

The highest ratings (3.7 or above) were given to encouragement of instructor, ease of access over web, discussion threads, checking grades, submitting assignments online, announcements feature, beneficial that software brings utilities together, and desire for use in future classes.  Lowest ratings (below 3.00) were provided for the group and personal web page features, and virtual chat.  These were also the features receiving the lowest response rate: only half the respondents even marked a rating.

Correlations were obtained between some of the variables on which we especially wanted to focus.  These correlations meet or exceed a p < .002 level of significance.

	Future Use	Satisfaction	Assists with Tasks
Ease of Use	.592	.723	.555
Future Use		.881	.663
Satisfaction			.787

There were no significant findings that previous experience with the product impacted ratings of satisfaction, ease of use, or desire for future use.

 

The following graphs illustrate the frequency of each rating level chosen by the respondents on selected dimensions of the survey (5 = high or positive). 

 

Some of the features in Blackboard are seen as more usable or useful than others. The chart below lists subjective ratings obtained from survey responses.  Applicability of certain features is somewhat dependent on their required or suggested use in any given course.  The instructions in the survey asked respondents to leave an item blank if they did not use the feature, thus, those that chose a rating had an opinion to express. 

Feature	Usability (percent choosing 4 or 5 on the rating scale)	Usefulness (percent choosing 4 or 5 on the rating scale)
Checking Grades	75.0%	84.6%
Announcement Feature	74.0%	82.7%
Submitting Assignments Online	62.5%	81.4%
Discussion Threads	65.2%	68.0%
Personal Web Page	47.1%	36.9%
Group Web Page	30.8%	37.5%
Virtual Chat	26.7%	29.4%

Discussion of Findings

It is interesting to see the mean averages from the survey results on the various options that Blackboard provides.  According to the survey, the most useful and usable item was the ability to check grades.  Yet, ironically this is one of the most problematic items for professors to implement within their class.  Other helpful features include the announcements section and the ability to submit assignments online. These features facilitate gaining information to complete tasks that the professor requests.  The students are then able to provide the completed assignment back to the professor for a grade.  This provides complete functionality of a class outside the physical classroom: just what Blackboard was looking for in providing a distance education software package.

The ratings of the students were above average on how satisfied they were with Blackboard, if they would use it again, and ease of using the interface.  This is promising for the software, but does not display any outstanding benefits for Blackboard either.  Even the most basic issues of password access caused some difficulty for students.

The lowest rated item for usefulness was the personal web page and the lowest rated item for usability was the group web page.  This is most likely due to the lack of perceived need to learn to use this particular feature in the present course.  Our view is that the usefulness and usability of these options may increase with use by distance education groups wanting to interact with their team members.  However, these functions could still be replaced by personal web pages and email communications from the group members using other, more familiar and less expensive software packages.  (More information on the mean values of the survey can be found in Appendix A.)

Participants most often commented on how confusing it was to determine which main category (left hand menu) a particular task fell under.  They wanted to see pop-up menus when they moved the mouse over the item.

Much of Blackboard’s potential success has to do with the extent to which professors incorporate it into their classes.  Blackboard has potential for considerable functionality.  It’s up to the circumstances of a class to make use of it.

APPENDIX A

Blackboard User Survey for MGT 500: Spring, 2000

The following questions ask about your experiences with using Blackboard CourseInfo. Please fill in your answers or use the 1-5 scale to circle your rating (with 5 meaning High or Positive).

(If you have not used a feature that is discussed within the question, please leave it blank)

Age:  28        Sex.   M   20   F  10      Part-time  6        Full-time  24

1.     Have you used Blackboard CourseInfo in a previous class?   Yes  16   No  14

2.     How many times per week, on average, did you access Blackboard CourseInfo?
Number of Times _1.25

3.     How encouraging was your instructor to use Blackboard?      3.8  1 ­ 2 ­ 3 ­ 4 ­ 5

4.     Please rate the ease of accessing the Blackboard software over the web browser?

3.9     1 ­ 2 ­ 3 ­ 4 ­ 5

5.     Have you experienced any difficulties in:

Password access                        Yes __21___             No ___9___

Browser                                      Yes ___2___             No __27___

Accessing URL                            Yes ___9___             No __20___

Error messages                           Yes __11___             No __18___

Other                                          Yes ___5___             No __19___

6.     Please rate both how Useful and Usable you find the following
Blackboard features

	Useful		Usable
Virtual chat	2.7	1 ­ 2 ­ 3 ­ 4 ­ 5	2.8
Discussion threads	3.8	1 ­ 2 ­ 3 ­ 4 ­ 5	3.4
Personal web page	2.8	1 ­ 2 ­ 3 ­ 4 ­ 5	3.2
Group web page	2.6	1 ­ 2 ­ 3 ­ 4 ­ 5	2.7
Checking grades	4.4	1 ­ 2 ­ 3 ­ 4 ­ 5	4.0
Submit assignments online	4.0	1 ­ 2 ­ 3 ­ 4 ­ 5	3.6
Announcements	4.2	1 ­ 2 ­ 3 ­ 4 ­ 5	4.1

 

7	How well does the layout of the interface allow you to find the tools you need to accomplish a task?	3.4    1 ­ 2 ­ 3 ­ 4 ­ 5
8	Does the software assist you with your tasks or assignments?	3.0    1 ­ 2 ­ 3 ­ 4 ­ 5
9	How would you rate the help screens for learning how to accomplish a task?	3.1    1 ­ 2 ­ 3 ­ 4 ­ 5
10	Do you feel the number of steps required to accomplish a given task is:	3.4    1 ­ 2 ­ 3 ­ 4 ­ 5
11	How beneficial to you is it that this software brings together various types of utilities in one place?	4.7    1 ­ 2 ­ 3 ­ 4 ­ 5
12	How would you rate the ease of use of the interface?	3.5    1 ­ 2 ­ 3 ­ 4 ­ 5
13	Would you like to use the Blackboard software in future classes?	3.7    1 ­ 2 ­ 3 ­ 4 ­ 5
14	What is your overall satisfaction with the Blackboard software?	3.5    1 ­ 2 ­ 3 ­ 4 ­ 5

 

APPENDIX B

Observation Record Sheet for Usability Testing

Blackboard CourseInfo Project

April 13, 2000

What parts of the interface were confusing, unclear, or hard to use?

What would you like to change?

What did you like about the interface?

Any other questions/comments?

APPENDIX C

Instructions for Evaluators

Explain Purpose

The Blackboard Project Team is studying how users interact with the Blackboard CourseInfo software interface.  We want to see what issues may be encountered.  Software designers can benefit greatly from this information, and the company can improve its product.

Explain Voluntary Participation

Thank you for volunteering to help us evaluate this product.  You are free to leave the session at any time if you wish.

Explain Your Role and Tasks

We will give you an instruction sheet explaining each task you will be asked to do.  While you are performing the tasks, we will be making notes on the process.  Be assured that we are not judging your performance, but are looking at how users interact with the interface. Our role will be to observe with minimal interaction, and thus will not tell you how to do a task.  However, if a circumstance arises that prevents you from being able to continue, we will assist.

Explain That Participants Should Think Aloud

Part of the benefits we gain will come from your comments as you work through the exercises.  You are strongly encouraged to think aloud as you go and verbally explain what you’re doing, what questions come to mind, and express any frustrations or satisfactions you feel.

Questions

Do you have any questions before we start?

Give Instruction Sheet - Mark the Observation Record Sheet as appropriate

At the Conclusion

(Ask follow-up questions listed on the recording sheet.)

Do you have any additional comments or questions? 

Thank you for your time and efforts.

APPENDIX D

Written Instructions for Test Volunteers

At your computer, start Internet Explorer or Netscape and enter the following URL:

http://www.bsu.edu/educate/   (choose your MGT 500 class)

When asked for your username, enter your VAX username
(enter lowercase letters)

Your password is your BSU student ID number (no dashes)

Try to accomplish each of the tasks listed below.  As you accomplish each item, let us know.  Remember that it is helpful for us that you think aloud as you go.

List of tasks:

1.  Bring up the lecture slides for the topic on Motivation.

2.  Using the CourseInfo system, send an email to a single user in your class.

3.  If your professor decided to post grades on Blackboard, show how you look them up.

4.  By default, Blackboard uses your VAX address as your email address.  Change this information on the system to an email account you currently use.  (If you have already changed your information or you want your VAX address to remain on the system, just show us how you would perform such a task.)

5.  Go to the discussion board and add an item or respond to someone else’s item.

When you are finished with the task list, log out completely.

We will now ask you a few questions.

Thank you for your time and efforts.

Bibliography

Blackboard Inc. (2000).  [Online]  Bringing Education Online.  http://www.blackboard.com, September 24, 1999

Fredrickson, Scott.  (1999). “Untangling a Tangled Web: An Overview of Web-based Instruction Programs.  [T.H.E. Journal], June 1999.

Keating, Anne B. with Joseph Hargitai.  (1999.)  The Wired Professor: A Guide to Incorporating the World Wide Web in College Instruction.

Landauer, Thomas K.  (1995.) The Trouble with Computers: Usefulness, Usability, and Productivity.  Cambridge, MA: MIT, pp. 205-322.

Nielsen, Jakob. (2000, March 19).  [Online] Why You Only Need to Test With 5 Users. Alertbox. http://www.useit.com, April 14, 2000.

Shneiderman, Ben. (1998). Designing the User Interface Strategies for Effective Human-Computer Interaction. 3^rd Ed. Reading, MA: Addison Wesley, pp. 123-154.

“Usability Testing.”  AT&T Best Practices, Usability Engineering, Issue 1, June 1995, pp. 79-91.

“User Observation: Guidelines for Apple Developers.” (1988).  Human Interface Update, #11, pp. 1-8.

About the Authors:

Dr. Jay Gillette is Professor of Information and Communication Sciences at Ball State University’s Center for Information and Communication Sciences in Muncie, Indiana.  Dr. Gillette teaches and conducts research in human communication; information networking design and development; telecommunications regulation, public policy and economics; leadership and management for the Information Renaissance; and information theory.

As Professor and Associate Chair of the Department of Information Networking and Telecommunications at Fort Hays State University in Kansas, he helped set up the nation’s first undergraduate degree in this new communication field. He assisted in developing Carnegie Mellon University’s graduate degree in Information Networking. Dr. Gillette served as a Senior Policy Fellow at the Docking Institute of Public Affairs in Kansas, and as a Senior Fellow of Information Technology and Telecommunications at the Center for the New West in Colorado, the policy research institute sponsored by U S WEST. He can be contacted by phone at 765.285.3285, Fax 765.285.1516 and email <Jgillett@gw.bsu.edu>

Cindy Ruman is an Application Support Consultant with Ontario Systems, a developer of receivables management software located in Muncie, Indiana.  She provides customer support for a complex system solution, assisting clients who use technology products to meet their business needs. On occasion, this service includes training on portions of the software.

Ms. Ruman received a Bachelor of General Studies in Anthropology and Classical Studies in 1992 and a Master of Science in Information and Communication Sciences from Ball State University in May of 2000.   During her graduate studies she became involved in studying applications to support distance learning, specifically, serving as project manager of a group researching Blackboard CourseInfo.

---