JURIST - Fitzgerald: A Brief Comparison of "Courseware" for Exams or Self-Assessment Exercises on the Web

Lessons | Talkback | Archive
ЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧ
In this monthly column, law professors comment on the many academic opportunities and challenges presented by Web technology.

As with all JURIST columns, you're invited to Talkback. This month...
ЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧ
A Brief Comparison of "Courseware" for Exams or Self-Assessment Exercises on the Web

Peter L. Fitzgerald, Stetson University College of Law

It's that time of year again, the Examination Period!

This year, when my students asked if there was a "model" exam they could use to help prepare themselves for their finals, rather than simply pass out a copy of an old exam I decided to use the web to try and make the old exams into a more interactive learning experience. Accordingly, using two different software packages, I placed sample exams for both my International Law and International Trade Regulation courses online, along with answers and additional feedback to turn them into self-assessment exercises.

There are a variety of tools for developing web based teaching materials. Several are described in previous articles in Lessons on the Web. Other tools, including software packages specifically aimed at assisting in the development of "courseware," can be found by searching for product reviews on the internet. For example, one group of online or distance learning consultants, who call themselves the "The University of the Future," recently published a comparative review of six major full featured courseware development products. They considered Blackboard's CourseInfo, IntraLearn Software's Intralearn, MadDuck Technologies Web Course in a Box (which is also the basis of the Lexis Virtual Classroom), WebCT's product with the same name, WBT System's TopClass, and Virtual Learning Environment's VirtualU. While some of these commercial packages are targeted at the academic community, others seem more appropriate (and more appropriately priced) for corporate training programs.

I chose the new CaliAuthor program and the standard edition of MicroMedium's Trainer5 for my foray into this area, in an effort to sample both an "academic" and a reasonably priced "corporate" product. Both programs offer a number of features and capabilities which I did not explore. Moreover, I am sure that more proficient users will quickly be able to point out that many of the issues I encountered are more attributable to "operator error" rather than software design problems. You can assess the end results by looking at the International Law exam, prepared with Trainer5, and the International Trade Regulation exam, prepared with CaliAuthor.

Both provide examples of a typical three hour law school exam, including the standard instructions provided with each exam at our institution, and a series of multiple choice, true/false, and essay questions. Also, I've used the features available in the software to provide the answers or other feedback for each question. In the case of the multiple choice or true/false questions, the correct or incorrect answers are specifically noted. In the case of the essay questions, both a "low scoring" and a "high scoring" sample answer are provided for each essay question, based upon actual student responses to the old exams. Obviously, I am only using a limited portion of the two software packages' capabilities, the "exam" or "question-like" functions, rather than building a complete instruction module. Even so, it was evident that the process of creating a complete substantive teaching module would involve essentially the same steps and techniques as employed in putting these exams "online."

The most obvious difference between the two is that the exam created with Trainer5 is in a separate file which must be downloaded to the student's computer in order to run, whereas the exam created with CaliAuthor may be run directly from the course web page. The Trainer5 file is, however, entirely self-contained and no additional software is required for it to run once it is downloaded. Nevertheless, Trainer5 produced a fairly sizable file, on the order of 10 megabytes in the case of my International Law exam. This has both advantages and disadvantages.

Downloading large files over the internet can be time consuming, particularly if a slow connection is involved. Now that read-writeable CD drives are increasingly affordable, if the material were distributed to the students on a CD this "download" problem would be avoided, and the fact that Trainer5 runs as a self-contained file would be an advantage. The exam created with CaliAuthor could also be distributed on a CD, as well as running directly on the web, but doing so is a little more complicated than with Trainer5. The Cali software initially creates files, which Cali terms "books," that must be run under the Cali porgram. Unlike the files created by Trainer5, these "books" are not self-contained and the Cali software program would also be needed to run the file on the CD. Of course, the Cali software program could also be included along with the "book" on any CD which is prepared, or the students could use any other licensed copy of the Cali program which they might already have to run the file from the CD. Alternatively, if you wish to run the file or "book" on the web, as I did with the International Trade Regulation exam, the original file is run through an internet "wizard" and converted into a javascript file. After this conversion, the file runs in an ordinary web browser and the Cali software program is no longer needed.

The operation of the exams created with these two programs also differs. The Trainer5 file is more "full featured" and you (literally) have the capability of using more bells and whistles when the program runs. For example, the "answers" to the multiple choice and true/false questions in the International Law exam are all provided audibly. When a student picks an answer, a computer voice tells them whether they have selected a correct or incorrect response. A number of different "correct" and "incorrect" audio responses are provided with the Trainer5 software package, and were simply tied to the buttons representing the various answers to the questions much as sound clips are used in Powerpoint or other similar programs. Trainer5 also has an interesting "fill in the blank" short answer format, where the "correct" feedback is given to the student only when an exact match to the predetermined answer is provided. Additionally, the program also includes the ability to run audio or video clips automatically, or after the student spends a preset amount of time on a given "page." You can see this on the "welcome" and "congratulations" pages used to open and close the International Law exam.

CaliAuthor's multimedia capabilities are more limited. While it can also run audio and video clips, they have to be tied to a button which must be "clicked" to start the clip. Additionally, CaliAuthor does not come with a package of multimedia clips to use, although audio and video files obtained elsewhere are easily imported. Accordingly, the International Trade Regulation exam I converted with CaliAuthor to run on the web relies on textual feedback, rather than audio cues. When the student selects an "answer," a small text window pops up over the page. The pop-up windows are also color-coded, green for correct answers, red for incorrect, and a yellow window is also available for something in between. Simultaneously, a green "" or red "X" appears next to the student's selection on the question page. The messages in the pop-up text windows for each response are set when you create the file. In the International Trade Regulation exam, for example, you will see that I've used this capability to briefly elaborate on why a particular selection may be right or wrong, or to encourage the student to select some other response. In this exam, the "low scoring" and "high scoring" sample essay answers also appear in pop-up text windows. There is also a button in the menu bar, labeled "answer," which the students can press to be shown the correct response(s) if they can't figure them out for themselves. This is particularly useful for "check the box" questions which can accommodate multiple correct responses. Clicking the "answer" button without selecting a response generates a special alert window, which reads "Gee whiz, you should try it at least once!"

Both programs are packaged with good tutorials instructing users how to use the software. Trainer5 also comes with a hardcopy manual, but some of the Trainer5 documentation does not appear to be up to date. For example, contrary to what is stated in the manual, when using the "fill in the blank" question form the software does not currently permit you to use wildcard characters. This effectively limits "fill in the blank"questions to those where you can reasonably anticipate the exact words or phrases the students will provide in their answers. MicroMedium's support was fairly responsive to email, but I felt their "frequently asked questions" web page for Trainer5 could be more extensive. Additionally, there are so many features available in the Trainer5 program that it is easy to think it can do more than it does. This can lead to problems. For example, until I realized you could not "cut and paste" elements such as button designs from among the various prepared templates which come with the software, I experienced numerous software crashes. One crash completely removed the Trainer5 file I had been working on from all file lists or directories on my computer, and required the use of a commercial "undelete" program to recover it and retrieve the material I had already created. The end product, however, is a "slicker" stand alone module than that produced by CaliAuthor. Incidentally, the "professional" version of Trainer5 has numerous additional features, such as a potentially very useful ability to automatically score and forward email reports regarding the students' answers, but it also costs more than four times as much as the "standard" edition.

CaliAuthor's more limited multimedia functionality is a little less flashy, but it has the distinct advantage of being able to create files which will run in a web browser. Its tutorial provides much more of a "step by step" guide to creating files or "books," and it was less likely to crash. While there is an email support list which appears to be very quiet and less responsive than with a commercial product like Trainer5, there are very good materials on its web site for authoring and technical support. The internet "wizard" which converts CaliAuthor's "books" into code which will run in a web browser is extremely easy to use. However, it does imbed all of the text into javascript commands. This makes it is harder to edit the appearance of the resulting web pages with something like Microsoft's FrontPage. For example, to adjust the placement or typeface of selected portions of the text, you need to view the actual html code and manually insert the appropriate html tags within the javascript the wizard created. None of this will appear in FrontPage's "normal" view where most editing is usually done, although the actual web pages CaliAuthor produces will be properly displayed in the FrontPage "preview" window. Of course, this only concerns those who might want to change the basic web pages the wizard creates. Overall, perhaps because of its less ambitious technical design, my subjective sense was that CaliAuthor was much more intuitive and easier to use than Trainer5.

Of course, the real question is how these self-assessment exams were received by students who grew up with MTV and video games? In a word "great"! The students, even those who were not particularly computer literate, uniformly enjoyed the ability to test themselves against these computerized exams. The ability to get immediate feedback as to whether their answers were correct or incorrect was something they appreciated. There was a decided preference for the International Trade Regulation exam created with CaliAuthor, however, for two basic reasons. Firstly, the ability to run the exam in a web browser rather than having to download a file was seen as a major convenience. Secondly, the audio feedback to the multiple choice or true/false questions in the International Law exam merely indicated which answers were correct or incorrect. With the pop-up text windows in the International Trade Regulation exam I included more information as to why a particular response was correct or incorrect, which helped make the exam into a more useful tool for the students. In other words, and unsurprisingly, content won out over multimedia bells and whistles. The Trainer5 software clearly would permit you to do the same thing and link a particular response to another page where additional textual explanation could similarly be provided. However, since the CaliAuthor design depends upon these pop-up windows as the primary feedback tool for the students' answers to questions, the program's basic format makes it easier to do than with the more full featured Trainer5.

As with any bit of new software, there are also the odd issues which arise when folks simply try to run the program without carefully reading the accompanying material. With the International Law exam created with Trainer5, despite indicating on the course syllabus that the sound on one's computer needs to be turned "on" in order to hear the answers, more than one student failed to do so and came to me to ask for the answers to the multiple choice and true/false questions. The obvious design point here was that I should have placed that particular instruction at the beginning of the exam file itself, rather than on the web page from which the file was to be downloaded. More startling, however, were the students who asked if they could see one of my exams for the International Trade Regulation course rather than something from Cali! The CaliAuthor internet "wizard" creates a coverpage and an end page which indicates that the module is created with CaliAuthor, references the Cali license agreements, and even provides for email feedback to Cali. A couple of students evidently took this to mean that this was some "off the shelf" self-assessment exercise produced elsewhere (perhaps also reflecting their experience that other Cali courses were not prepared "locally"), even though I was clearly identified as the author. I believe this problem was largely a matter of emphasis and layout. If the javascript created by the CaliAuthor internet "wizard" were more easily edited, this could easily be fixed by individual authors adjusting the text or typeface used on these pages.

Although my survey of software for computer assisted instruction and self-testing was far from complete, I found that the programs offer more than the expected ability to "computerize" a rigid set of questions with predefined multiple choice answers. Both of the programs I considered, and presumably many of the other programs which are available as well, enable authors to develop their materials in a way which can accommodate not only traditional "objective" questions, but which also permit multiple types of formats such as matching, ordering, sliding scale, and short answer questions which I did not explore fully. Moreover, the ability to link further materials to any particular response means that, with some forethought, it's possible to introduce a fair degree of nuance into any given question and answer set. The technology is clearly still not at the point where some artificial intelligence can grade a traditional law school essay question, but there is more capability available in these programs than one might imagine. They do require some investment of time, but not substantially more than that required to learn other tools such as PowerPoint. Most important, however, is that students seem to find the self-assessment exams created with these tools to be valuable adjuncts to the learning process in law school, enabling them to test their knowledge and ability to apply the various concepts discussed in the classroom on their own schedule and at their own pace.

© 1999 by Peter Fitzgerald. All rights reserved.
ЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧ
The views expressed in this column are solely those of its author, and do not reflect those of JURIST, its Advisory Board, its staff or its host institutions.
ЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧ

Engaged? Enraged? JURIST would like to hear your reactions to this column and the issues it raises... ЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧЧ