Está en la página 1de 8

The Walden’s Paths Quiz Engine

Avital Arora, Emily Barker, Unmil P. Karadkar, Pratik Dave, Luis Francisco-Revilla, Richard Furuta,
Frank Shipman, Suvendu Dash, Zubin Dalal
Center for the Study of Digital Libraries and Department of Computer Science
Texas A&M University
College Station, TX 77843-3112, USA
Phone: +1-979-845-3839
{avital, eluke, unmil, pratik, l0f0954, furuta, shipman, suvendud, zubin}

Abstract: The Walden’s Paths project is developing tools to facilitate the inclusion of Web-based
materials in classroom education. It currently includes components to create, present and manage
paths that recontextualize information on the Web. In this paper we present the Quiz Engine, a full-
featured online testing system that extends the repertoire of our tools to include Web-based
assessment of the knowledge gained by the students. It allows teachers to author the different types
of questions commonly used in paper-based testing and generates interactive as well as passive
tests with various characteristics. It is designed to support teachers with little experience in Web
technologies to create and administer online tests in a variety of settings for aiding students’ self-
assessment while learning or for administering graded examinations.

Periodic evaluation is essential to encourage students to acquire and apply the knowledge, information and
the skills that they learn. Evaluation helps students understand their strengths and weaknesses and helps teachers
assess the effectiveness of their teaching methods and focus their attention on areas where the students need
improvement. Teachers employ various evaluation mechanisms to verify that students are gaining the required
proficiency: they administer tests and quizzes, assign homework, lab work and involve students in projects and
demonstrations that apply the knowledge and skills gained.
Online tests and quizzes with automated grading mechanisms (Browne 2002, Hall et al. 1999, Woit &
Mason 2000) minimize the time needed for grading and can provide quick feedback for the students’ benefit.
Meaningful and timely response is of utmost importance if students are to benefit from the evaluation (Browne 2002,
Flewelling et al. 2001). Adaptive online assessment mechanisms like the QUIZIT system (Tinoco et al. 1997) tailor
the difficulty level of the quizzes to maximize the benefits for individual students. Web-based testing systems are
more accessible to students as they transcend the space and time barriers imposed by most conventional testing
mechanisms. Students may take practice tests at their convenience from a location and at a time of their choice.
Martinez stresses that the Web offers the perfect technology and environment for precision learning because learners
can be uniquely identified, content can be personalized for them individually, and subsequent progress can be
monitored, supported, reported and assessed cost effectively (Martinez 2002). Research suggests that online testing
is an effective mechanism for evaluating student knowledge of the subject matter and that there is a good correlation
between the scores obtained by students on paper-based and online tests (Kumar 1999). Online testing is also an
effective preparatory mechanism for in-class tests (Hall et al. 1999).
In spite of the advantages of online evaluation many teachers are reluctant to take the plunge, mostly
because they are unfamiliar with the technology that goes into creating and managing these systems (Arif, 2001,
Matuga 2001). The process of setting up online assessment and evaluation activities is completely different from
that followed in a traditional classroom setting. Commercially available software such as WebCT (WebCT 2003)
and Blackboard (Blackboard 2003) support teachers in creating, managing and hosting online courses by integrating
the syllabi, multimedia course content and assignments and providing other allied services.
Walden’s Paths (Shipman et al. 1999), our suite of tools, supports teachers with varied technological
backgrounds in tailoring Web-based information for classroom use. The Web is a vast, unedited information
repository with little publishing control and contains information that may be inaccurate or inappropriate for use in
an educational setting. Even when correct, Web materials are often authored for purposes other than education and
can leave students confused about the context and vocabulary. Walden’s Paths is motivated by the belief that the
material already available on the Web can be recontextualized for use in educational settings. It provides teachers a
way to focus student exploration, to add explanation and context to materials for use in the classroom environment.
The next section of this paper provides a brief overview of Walden’s Paths. We then present our motivation
and rationale for the quiz system, followed by the design and operational overview. We conclude with a discussion
of the various issues that we plan to address in the near future to enhance the value of the Quiz Engine.

Walden’s Paths
Walden’s Paths ( supports the reuse of Web-based information via the
metaphor of a linear, guided path. Guided paths were envisioned as far back as Vannevar Bush’s Memex (Bush
1945). Systems implementing guided paths have been investigated with limited user communities since the 1980s
(Trigg 1988, Zellweger 1989). Walden’s Paths is a suite of tools that supports creation, presentation and
maintenance (Francisco-Revilla et al. 2001) of linear paths over Web-based materials.
Teachers (or more generally, path authors) create paths by searching for relevant Web pages and adding the
ones that they find interesting into the authoring interface. They may then add text or HTML annotations to reflect
their perspective on the topic addressed by each included page. These annotations may elucidate the page content,
highlight points of interest, provide instructions to students or ask questions that the students must answer.
Students (or more generally, path readers) view the paths from any of the standard Web browsers via the
Path Server. Path readers typically begin browsing by choosing a path from a list presented to them. Figure 1
illustrates the navigation interface while browsing a path. The Path Server displays the Web pages along with the
contextualizing annotations as organized by the teacher. The bottom part of the browser window displays the Web
page, as it would appear without the Path Server interface. The annotation added by the teacher is displayed in the
top-right part of the window. Readers navigate the path using controls displayed in the top-left portion. They may
view the pages on the path sequentially by clicking on the “Next” and “Back” images or jump to other pages on the
path by scrolling to and clicking on the “paw” corresponding to the position of the page. While following the path,
readers are free to follow links embedded in the Web pages to explore the surrounding information space. Following
these links takes the reader “off the path”. While the reader is browsing the information space off the path the
control widgets described above are replaced by a single “Back to path” image that links back to the page from
which the reader wandered off the path as shown in figure 2. This feature provides readers with an easy way to
return to the path once they are done exploring. Thus a student may browse through pages not included in the path,
secure in the knowledge that she can return to the path with a single click. By providing context to the vast resources
on the Web, the Path Server supports guided navigation as well as exploratory learning.

Figure 1: The Walden’s Path Server Figure 2: Browsing off the path
Our experience with Walden’s Paths has shown that soon after they start using the system, teachers and
students alike shift focus from the underlying metaphor of the path to the subject matter presented and the contents
of the Web pages (Furuta et al. 1997). We attribute this “fading” of the technology to the simplicity of the metaphor
and of the interface.

Motivation for a Developing a Quiz System

Some of the teachers who used Walden’s Paths early on annotated their paths with questions that the
students would answer upon reading each page. The annotations for the pages shown in figures 1 and 2 contain
questions of this type for a path that described the Kitt Peak Observatory and compared it to others. Students were
asked to turn in their answers to these questions on paper, which the instructor graded by hand. The teacher
successfully mixed digital and paper media to achieve his curricular goal of evaluating students’ understanding of
the material he had included in the path.
A quiz system with automated grading support frees teachers of the burden of grading some kinds of
questions, for example, multiple choice and true-or-false questions. It aids teachers in evaluating students’
understanding of the materials that they have learned from the paths and can provide instantaneous results and
immediate feedback. Teachers may present quizzes with randomly ordered questions to address issues of academic
dishonesty where students may peek at their neighbors’ monitors (Browne 2002). Over time, teachers may build a
large database of questions to serve as a question bank and quizzes may be generated by randomly picking questions
from this database. Thus, different (but comparable) tests can be generated for individual students to further
discourage cheating on tests. Various statistics and trends regarding students’ performance on the test may be
automatically compiled and emailed to the teacher. Teachers may easily analyze the results generated by the quiz to
identify problem areas and common sources of errors. The feedback obtained in the form of the students’
performance helps instructors analyze and improve various aspects of their teaching, such as the course contents,
style of delivery or the nature of interaction in the class. In the case of Walden’s Paths, such analysis may suggest
improvements to a path’s content. If a large number of students are found answering a particular question incorrectly
the instructor may go back and examine the path to see if the concept needs additional illustration or
A quiz system also aids students in fine-tuning their learning process. Students can choose to attempt
practice quizzes on their own initiative, to test how well they are prepared for an upcoming exam or to diagnose the
effectiveness of their study patterns. Practice quizzes, which may be modeled along the pattern of the exams, help
students prepare for their final examinations. Students may initiate a quiz as they are browsing a path to reassure
themselves that they have grasped the essence of the path (or of some pages on the path). The quiz system provides
immediate feedback, especially in quizzes taken for self-evaluation. The quiz system may randomly select questions
for a given quiz, thus exposing the students to a variety of questions on the same topic. As the database of questions
grows larger a quiz can be taken more than once to answer a possibly different set of questions.
The immediate feedback offered by the Walden’s Paths Quiz Engine will allow students to quickly identify
their mistakes and if necessary view the relevant path(s) for more information. Although graded paper tests provide
some feedback, the response is typically delayed and teachers rarely have the time to provide detailed explanations
to each student or to point them to additional resources individually. The students may or may not take the initiative
to find out what went wrong. The quiz mechanism allows teachers to provide a Web resource where students may
get additional information regarding the question. With immediate feedback and the availability of additional
resources students are more likely to view this information, hopefully resulting in better performance on the actual
The quiz facilitates easy accessibility since an online quiz removes distance barriers and caters to the
student's time commitments, location, preferences and convenience. Quizzes could also be used as the primary
evaluation mechanism for a course taught after hours, possibly for part-time students or in a distance learning class.
We conducted an informal survey of teachers in various academic disciplines at K-12 and university levels
to gain an insight into the factors that affect their decisions regarding use of online testing systems. Some of these
teachers use online testing extensively while others had little experience in this area. Most teachers surveyed use the
Web for disseminating course materials to their students. Some teachers noted that most tools do not support the
features required to answer their questions. For example, one teacher includes questions that require students to
draw graphs, write complex mathematical formulae and derive equations on her tests. While developing tools that
will support rich interactions is a long-term goal, we have chosen to begin by supporting the types of questions that
are most commonly used on tests in an effort to address the issues faced by a wider community of teachers.
Figure 3: The Question Authoring Interface Figure 4: The Quiz Generation Interface

The Walden’s Paths Quiz Engine

The Quiz Engine has been developed with simplicity of use as the key design principle with a view to
catering to audiences with varied technological expertise. The Quiz Engine is a generic quiz authoring and
management system. While designed to work with Walden’s Paths, it could be easily used to generate quizzes in
isolation, and presumably, with other software.
The process of generating quizzes using the Quiz Engine consists of two activities, authoring questions and
choosing questions to form tests. Authoring questions is a long-term process as teachers may add questions about
various paths over a period of time. These questions are stored in central database that acts as a repository for
generating quizzes for all teachers. When teachers need to generate quizzes they specify various criteria to select a
set of questions from the question database. They may customize a generated quiz by adding and deleting specific
questions. The Quiz Engine saves this quiz and returns a URL for it. A teacher may send the URL to her students for
taking the quiz or publicize the links for practice quizzes on her class Website. By separating the process of creating
questions from that of combining these to form tests, the Quiz Engine allows teachers more flexibility in generating
quizzes as quizzes created for a path at any time include questions from the pool of all questions that are currently
The authoring interface displayed in figure 3 allows teachers to enter questions into the database. A teacher
begins by identifying the path and the page(s) containing the material to which this question refers. She types the
question and categorizes it as either a general question—a question regarding general principles related to the topic
of the path—or a topic-specific question—one that deals with details specific to a topic or a sub-topic within the
path. The ability to distinguish between these two categories is useful during quiz generation. She then decides upon
the format of the question. The Quiz Engine currently supports questions of four types:
questions with a specific word or a specific phrase as the correct answer
multiple-choice questions, with a single correct answer
multiple-choice questions, with multiple correct answers
questions with longer, essay-type answers
The Quiz Engine is capable of automatically evaluating questions in the first three formats. For the multiple-
choice questions, the teacher specifies the various options and indicates which of these are the correct answers.
True-or-false questions are treated as instances of multiple-choice questions with one correct answer. Answers to
essay-type answers are e-mailed to the teacher for evaluation and must be manually graded. She assigns a difficulty
level between 1 (easy) to 5 (difficult) to each question. This enables her to form quizzes of varying composition of
easy or challenging questions. The teacher may provide a URL to an image or an illustrative figure as a part of the
question. The image may be an integral part of the question or it may act as a clue to the answer. For example, the
question “Name the person shown in the accompanying figure” requires that there be an image associated with it.
On the other hand, the fill-in-the-blank question, “_____ live in floating lodges.”, may be illustrated with a picture
of a beaver to provide students with a visual clue. The teacher may also include a URL that provides additional
information about the topic of the question.
Figure 5: The Interactive (short) Quiz Figure 6: The Passive (long) Quiz

The Quiz Engine validates the teacher’s choices and alerts her if invalid data is entered. Informative tool
tips associated with the various input fields give her a better idea about the expected data for each field. The teacher
may preview the data entered to ensure accuracy before it is added to the question database and edit the added
questions using an interface similar to that used for authoring.
To create a quiz, the teacher invokes the Quiz Generation Interface illustrated in figure 4. This interface
accepts her preferences and uses these to generate a draft proposal for the quiz. The teacher specifies the path for
which she is generating the quiz and names it for easy identification. She then enters the number of questions she
would like to include in the draft quiz. The size of the final quiz may vary as teacher may add or delete questions
from the system generated draft. The Quiz Engine e-mails the student responses to the teacher if she so prefers, the
most common case of this may be a quiz that includes questions with long answers. The teacher may choose to
randomize the order in which questions are displayed to the students as they take the test. While all students answer
the same questions, this makes it considerably more difficult to copy the answers from nearby students’ computer
displays. She may also choose to prioritize questions that least recently used questions to ensure that all questions in
her question bank are used.
Quizzes to be used for grading require that students authenticate themselves, while the practice quizzes
may be taken anonymously. Teachers may also want students to identify themselves if the Quiz Engine is
monitoring the progress of individual students to tailor questions at the appropriate difficulty level. The current
prototype does not monitor students’ progress on practice tests.
The teacher may broaden the scope of the quiz by including questions authored by other teachers. This
feature allows teachers who teach different sections of a class to pool resources or to include questions authored by
another teacher who has taught the class earlier.
A quiz can be tailored further to focus on the general or specific concepts by including questions from
various categories described earlier in desired proportions. A quiz that focuses on the general principles may include
a higher percentage of overview questions, while a quiz that expects students to know details may consist mostly of
sub-topic-specific questions. The Quiz Engine returns a draft quiz based on the teacher’s preferences. She may then
preen the quiz by deleting individual questions and adding new ones.
The quiz may be either interactive or passive. As shown in figure 5, interactive quizzes display one
question at a time. These quizzes evaluate the student input for each question and display the result with any
suggestions or additional information provided by the teacher before continuing to the next question. As one can
guess, interactive quizzes are best used as a tool for self-assessment or as preparatory tests. It is difficult to impose
time constraints as the time taken to complete these quizzes varies depending upon whether they view the additional
information provided by the teacher. During the quiz, students may also access an overview page that displays the
result of all questions they have answered so far. A passive quiz, as shown in figure 6, displays all questions at a
time and students may answer the questions in any order they choose. All questions are graded after the student
indicates that he has finished answering the quiz and he receives no feedback while progressing through the
questions. The additional information entered by the teacher is not displayed either. These quizzes are the digital
analogues of in class, closed book, time delimited exams. Indeed, a teacher may print out this quiz and administer it
as a paper-based test.
Discussion and Future Work
With educational practices and tools changing at a great pace, Nichols (Nichols 2002) discusses how the
old dichotomy of traditional face-to-face and distance education is becoming obsolete as more diverse mixed-mode
forms of education delivery emerge. Teachers who teach a class in the classroom often make course notes or
discussions available over the Web. They may use e-mail as the primary communication tool outside the classroom
or may use Web-based discussion groups to encourage curricular discussion beyond class hours. Technology can be
used to foster a deeper approach to learning by actively engaging students in the learning process and thus placing
the focus and responsibility for learning upon the students themselves (Goodwin et al. 2001). As the responsibility
shifts to the students, they must also be empowered to successfully fulfill their responsibilities. Currently, the
students may only take quizzes that are created by the teachers. However, motivated students may wish to track their
progress more frequently or at a finer grain than that of the path. Providing support for student-created custom
quizzes may engage them better in the learning process. For example, a student may generate a quiz to test his grasp
of the pages that he has visited so far on the current path. Students may invoke quiz sessions while navigating along
a path, possibly by clicking on a “Take Quiz Now” button, to generate a quiz on the pages that they have viewed so
far. Providing support for such features requires that the Path Server be augmented to record the student sessions and
pass it to the Quiz Engine for generating the quiz.
Integrating the Path Server and the Quiz Engine may also help change the nature of the paths themselves.
Teachers may add question pages as a part of their paths. These pages may include a random or a specific question
relating to the materials that they have seen so far. These pages have the potential to introduce a wide variety in
interaction while reading the paths. A question page may announce itself to students by being visually different from
information pages or it may masquerade as a regular path page and surprise the student when it is viewed.
Answering the question may be mandatory or optional and answering a question incorrectly could redirect the
student to the information page that contains the correct answer. Teachers could use surprise question pages to
counter the issue of students being easily distracted during online learning sessions, a problem identified by
Bergstrom (Bergstrom 1997). Ensuring that students actually complete their assigned readings and tasks is another
important problem that must be addressed (Mason & Woit 1998). In an effort to ensure that students are indeed
paying attention, teachers may grade the answers to these surprise questions to simulate pop quizzes over the paths
and thus enhance the online learning experience.
As quiz questions are closely related to contents of pages included in the paths, teachers may find it easier
to author questions for the quiz while creating the paths. While they may add questions to existing paths, the co-
authoring of paths and quiz questions will help strengthen the narratives of the paths as well the variety of questions
for a more in-depth assessment.
The current system ties every question to a specific page on the given path. However, it is easy to imagine
more complex relationships between questions and pages. An overview question may relate to concepts expressed in
the path as a whole or to a part of the path (a few pages on the path). It is essential to remember that students may
not always view the pages serially by their ordering in the path. Answers to some questions may vary depending
upon the order in which a student viewed the pages. For example, a teacher may create a path consisting of
individual images that may convey different impressions depending upon the order of their viewing. In this case, the
teacher may ask questions the answers to which may vary depending upon the order in which the pages are viewed.
They may also author several questions that may only be asked if a student views the pages in a specific order.
Providing a simple interface that supports expression of complex relationships between questions and the materials
presented by the paths is a difficult task. While technologically savvy teachers may not flinch from expressing
intricate relationships via complex conditional expressions, providing a simple interface with lesser expressivity will
be more useful to a larger community of teachers.
We are also striving to support a wider variety of questions. Some types of questions that are commonly
used in paper-based testing environment are not suitable for online testing (Arif 2001), for example, questions with
long, essay-type answers. We provide a work around by e-mailing the answer to the teacher, who must grade it
manually. While the teachers may accept this system for tests that are graded, they may find it extremely
inconvenient to receive email for each long question answered during self-assessment quizzes.
The reluctance to adopt online testing mechanism stems from various factors. Teachers are accustomed to
administering paper-based tests and may not be unwilling to migrate as it implies learning new techniques and
technology that they are unfamiliar with. Teachers may not be able to replicate all question types in paper-based
settings in online testing environments. While it lacks some features of paper-based testing, the online systems make
up by providing automated grading, ease of analyzing student responses to detect patterns and student-specific
adaptation for effective learning support. Providing appropriate social as well as technological solutions will
encourage teachers and students to incorporate online testing in their educational environments.
An online assessment system that ties in with the Walden’s Paths system can serve multiple advantages to
both, the students and the teachers. With the addition of the Quiz Engine, Walden’s Paths can be used for delivering
curricular content, engaging the learner in the topic at hand, facilitating, even forcing, learning and assessing the
knowledge gained. The Quiz Engine may work as an alternative to or as a support tool for traditional paper-based
quizzes. The design and development of the tool has helped us gain a perspective of the complex processes
underlying effective student evaluation. We are channeling the understanding gained from the development of the
initial prototype into addressing these issues via social and technological solutions.

Arif, A. (2001). Learning from the Web: Are Students Ready or Not?. In the Journal of International Forum of Educational
Technology & Society and IEEE Learning Technology Task Force, Oct. 2001. (accessed April 25, 2003).
Bergstrom, S. (1997). Distance learning technology for aviation training. In Aviation Industry Computer Based Training
Committee working papers, June 1997. (accessed April 25, 2003).
Blackboard Inc. (2003). Blackboard Course Management System 5.1. Blackboard Inc. (accessed
April 25, 2003).
Browne, R. (2002). Automated Tutorial and Assignment Assessment. In the Journal of International Forum of Educational
Technology & Society and IEEE Learning Technology Task Force, Jan. 2002. (accessed April 25, 2003).
Bush, V. (1945). As We May Think. The Atlantic Monthly, Vol. 176, No. 1, July 1945. 101-108.
Flewelling, J. and Snider, D. (2001). CARLA (Computer-Assisted Recorded Language Assessment) – Software for Evaluating
Student Oral Responses, ICTE Educational Technology Resource Library.
Francisco-Revilla, L., Shipman, F., Furuta, R., Karadkar, U. and Arora, A. (2001). Managing Change on the Web. In
Proceedings of First ACM/IEEE-CS Joint Conference on Digital Libraries, June 2001, Roanoke, VA. 67-76.
Furuta, R., Shipman, F, Marshall, C., Brenner, D. and Hsieh, H. (1997). Hypertext Paths and the World-Wide Web: Experiences
with Walden’s Paths. In Proceedings of Hypertext ’97, April 1997, Southampton, UK. 167-176.
Goodwin, C., Graham, M. and Scarborough, H. (2001). Developing an asynchronous learning network. In the Journal of
International Forum of Educational Technology & Society and IEEE Learning Technology Task Force, Oct. 2001. (accessed April 25, 2003).
Hall, R., Pilant, M. and Strader, A. (1999). The Impact of Web-based Instruction in an Applied Statistics Course. In Proceedings
of the International Conference on Mathematics/Science Education and Technology (M/SET 99), Mar. 1999. (accessed April 25, 2003).
Kumar, A. (1999). On changing from written to online tests in Computer Science I: an assessment. In ACM SIGCSE Bulletin,
Proceedings of the 4th Annual SIGCSE/SIGCUE on Innovation and Technology in Computer Science Education, vol. 31,
Sept. 1999. 25-28.
Matuga, M. (2001). Electronic Pedagogical Practice: The Art and Science of Teaching and Learning On-line. In the Journal of
International Forum of Educational Technology & Society and IEEE Learning Technology Task Force, July 2001. (accessed April 25, 2003).
Martinez, M. (2002). Beyond classroom solutions: new design perspectives for online learning excellence. Discussion Summaries
in the Journal of International Forum of Educational Technology & Society and IEEE Learning Technology Task Force, Jan.
2002. (accessed April 25, 2003).
Nichols, M. (2002). Principles of best practice for 21st century education. In Discussion Summaries in the Journal of
International Forum of Educational Technology & Society and IEEE Learning Technology Task Force, Apr. 2002. (accessed April 25, 2003).
Shipman, F., Furuta, R., Hsieh, H., Francisco-Revilla, L., Karadkar, U., Rele, A., Shenoy, G. and Brenner, D. (1999). Using the
Internet in the Classroom: Variety in the Use of Walden's Paths. In Proceedings of ED-MEDIA 99: World Conference on
Educational Multimedia, Hypermedia, and Telecommunications, June 1999, Seattle, WA. 335-340.
Tinoco, L., Fox, E. and Barnette, N. (1997). Online Evaluation in WWW-based Courseware. In the Proceedings of the 28th
SIGCSE technical symposium on Computer Science Education, San Jose, CA. 194-198.
Trigg, R.H. (1988). Guided Tours and Tabletops: Tools for Communicating in a Hypertext Environment. In ACM Transactions
on Office Information Systems, 6 (4). 398-414.
WebCT (2003). WebCT Course Management System 3.8. WebCT, Inc., Lynnfield, MA. (accessed April
25, 2003).
Woit, D. and Mason, D. (2000). Enhancing student learning through on-line quizzes. In Proceedings of the 31st SIGCSE
technical symposium on Computer Science Education, Austin, TX. 367-371.
Zellweger, P. T. (1989). Scripted Documents: A Hypertext Path Mechanism. In Proceedings of the ACM Hypertext 89
Conference, New York, NY. 1-26.


This material is based upon work supported by the National Science Foundation under grant numbers DUE-0085798,
DUE-0121527 and IIS-0219540.