Distance Learning Application Demo Preliminary User Test

Table of Contents

1. User Test Objective
2. User Test Method
3. Findings

1. User test scripts/questions
2. Distance Learning Application (DLA) User Test Result

II. User Test Method

I chose three educators to user test who are my colleagues and study educational technology. The reason I user tested my colleagues was that I was not allowed to interview SGI's potential customers. The test users had to be educators who know instructional technology yet are not SGI's potential customers. My colleagues know learning theories and new technology well enough in order to give me good feedback on designing a good learning environment. Besides, two of the test users were formerly a teacher. Besides these three test users, informal user test was conducted with three additional users.

The user tests were conducted using screenshots of the demo. Due to technical difficulties and other confidentiality issues, I could not use the actual demo for the user tests. I, however, presumed that screenshots were sufficient in order to show the current version of the demo since it does not have much dynamic elements. I also used a short video clip to show the synchronized video and slides feature. These screenshots and the video clip can be viewed here.

Each user test session started with demographic questions (see APPENDIX A) to learn about the user's background and his/her general opinion on distance education. After telling the user the objectives of the DLA project and the user test, I showed each screenshot, explaining where the page is located and what role it plays. The user was allowed to ask any question or say his/her opinion any time promptly although most of them said their opinions after my explanation was finished. The session was concluded with questions regarding course content, media used and evaluation methods. All the three sessions were videotaped for a recording purpose only.

III. Findings

For the design of the demo such as its appearance, the three users' opinion varied. However, many other issues they pointed out were very similar to each other in some sense (see APPENDIX B).

The following list shows some of the issues found out in user testing.

• A simple design is preferred especially since DLA is going to play the role of a template.
• The frame feature in HTML is a good way to provide index as well as content. However, it is important to give enough physical space for the content instead of using the same layout for everything.
• Synchronized video and slides can be confusing and distracting although it seems nice to some people.
• Depending on how the course is organized, it should have the calendar in more than one style.
• There are some terms used inappropriately. (e.g. Usage Help, Follow-up and Reply)
• News group posts and Additional Resources should be organized in a better way.
• Well-designed index and menu are important for convenient navigation.
• Class Grade Sheet and Study Question / Quiz reports need to be redesigned.
• What is necessary information?
• Teachers prefer a grid look for grade sheets.
• How to allow teachers to manage the data for all the students and all the course activities?
• The amount of text needs to be reduced.
• 'Numbers' is more important than graphs.
• Registration Area should not be managed by teachers.

APPENDIX B: Distance Learning Application User Test Result

Location: Big Tree computer classroom in CERAS at Stanford University
Number of test users: Three
Date: March 5, 1999
 
 

	User #1	User #2	User #3
Demographic Questions
What were you doing before?	Teaching English in high- and middle schools Instructional design of technology Joining Virtual High School (distance learning program)	Trainer  Leadership development	4th grade elementary school teacher
What type of technology did you use in teaching?	Internet (WWW & email) last year in classroom		In classroom Kid Pix, HyperStudio, etc. For science projects, writing projects Macs, PCs, video The school was a beta test site for NASA's study Used video for math news show Outside of classroom Computers for work sheet and grade book Research on the web (mainly looking for software)
Have you taken a distance education (DE) class?		No
Which do you prefer, taking a traditional classroom style class or a DE class?		Traditional classroom style
What are your concerns with DE?	Social aspects  Getting people know well back and forth	Likely to lack face-to-face contact	Face-to-face interaction  Body language  Important to see students’ face to determine what is in their mind  Immediate feedback
DLA demo – Student mode
Course home page	Index needs to be reorganized   Some help with navigation needed?  Maybe pop-down menu style index?	Nice to have office hours and grading policy all written
Video clip & Slide show	Video is too small  Synchronized video and slides is nice	Synchronized video and slides is really nice, with close-ups of what is going on in the slides	Synchronized video and slides is confusing  Eyes going back and forth between the video and the slides  Slides could be built into the video – focus more on one thing instead of two things  Video should be bigger
Calendar	Too small  Text below the calendar could be annoying later – could be included in the Help page	"Enter your start date" – calendar adopts to students pace, depending on when they started	Nice to have Back and Forward arrows for the previous and next months  Options to show all the months needed for the entire course (allow toggling to hide other months)
Usage Help	There should be Index at the top of the page with a link to the topics	At first it seemed that Help means ‘technical help’ or ‘support’ – maybe a better word?
News Group (Currently in the teacher mode)	Should hide replies  Needs to be moved into Student site  "Lotus Note" can be used?  When subject should be changed? Maybe change it every post  Using both "Reply" and "Follow-up" – use only one for consistency	Should have a few different topic areas – technical difficulties, type of computers, etc.  What happens when clicking on a username in news?  The following things should be covered  Follow up  Email to this person  Email to teacher  Email to other students	Make posting news part of the requirement to encourage discussions
Additional Resources	Should be under each lecture rather than having one for the entire course – it could get very long later	How easy to modify the list?  How often updated?  Maybe FAQ? – to show why these resources are relevant	Is there a way for teachers to add more resources anytime easily?
Look and feel	70’s kind of color	Having Index in the left frame is nice  Maybe should show just Lecture 1 – 10 only in Index  Maybe "drop-down" menu?  Nice to have all and everything in one page
DLA demo - Instructor mode
Instructor mode home page	Why is the description of each function not considered Help?	Nice to have the descriptions	The title of each description should also have a link to its actual page
Class Grade Sheet	Creating the sheet (chart) automatically is cool  Showing scores sideways is awkward   Grid looking is familiar to teachers  Should show date and the number of times the students took the quiz  Maybe using Week1, Week2, Week3 method?  Does the quiz have to be taken by a certain date?  Allow teachers to click on and choose one date – manipulate date  Teachers want to see all date at once, or see either column or row at a time	Is this a packaged class?  Sometimes teachers want to have their own category (attendance, etc.)  Optional area for teachers to add to the grade sheet – personalizing	Want grid  Should show total points  Should show how many points the students got out of how many  Should be able to choose one test to view full information (just like they can view a student's full report)
Study Question / Quiz	"Quiz 1 was taken 9 times" – does this mean by one student, by 9 different students, or by 5 students?  Index at the top of the page to jump to the bottom is good  Too much text  Not enough space between different topics  Color should be used wisely to show correct/wrong answers Bar Graph Why ‘0’ has a bar   Not sure if a graph is necessary  Table is better than graph  Bars look clickable  Pie chart instead? – showing relative to class is better than to nothing Table at the bottom "Mean" is nice	"Study question" sounds like homework  Nice to have Index at the top of the page  "Mean" – too many digits	Knowing the number of the students who answered a question is nice  The information at the bottom of the page (such as mean) is nice Bar graph Hard number is better than graph (even for elementary school)  At least bar graph should be flipped (to vertical bars)  Should indicate how many out of all the students got this question right in class
Student Tracker	The number of the login times is good  Raw data (of student's login status) is nice to have – probably teachers won't use it at the beginning, but should be valuable later  In this course, it seems there are no interactions between students and teachers  If this is a ‘do it yourself’ course, Tracker is NOT necessary  If this is a real course, teachers will check Tracker and email students to say, "Hey. What's the deal here?" Index at the top of the page Are they referring all the students or just one? The chart at the bottom Confusing  Data and score are necessary  Showing scores vertically is better than horizontally	What is most important information and what is supporting information?  Maybe there is a different way to display this information?  Coloring?	Want to know where students are spending their time more than how frequently they look at material  Want to know if they are spending more time on video or text (on the same page)  Want to know how much time students spend on a specific question, if possible  Probably knowing when students study is not as important as how many times  Want to know where they are having their focus on  The chart has too much information – overloading teachers The chart at the bottom Confusing  Should highlight correct answers  Reduce text  Want to know how many questions they answered right  Want to know how many times they took Study Question
News Group Admin	"Remove Files" sounds like that it might remove the whole class. Sounds scary. Use another word such as "Remove news group posts"  Password in the Remove page – only for News Admin or for the whole course? If for the whole course, why does a teacher need to enter a password here?  If a list of the features that teachers can use in News Group Admin is displayed later on another page, why do they have to go through the previous page?	Should have a feature to sort by date, author, and subject  What does ‘single’ and ‘thread’ mean? – not clear
Registration Area	"Login name" and "Password" – are they for a student or for a teacher?   It is a problem that a teacher needs to know a student's password or he needs to be with the teacher physically  Can teacher's password override a student's?	Why do teachers need to register students?  "case sensitive" password – people might forget case	It is students’ responsibility to register themselves, especially at university level  Teacher should just gets a list of the students in class  Maybe better to have "Status Section"? – for teachers to view any corrections that students have made and maybe to view the reason why they are taking this course, etc.  Registration task should not be teachers’ responsibility
Other Questions
Can you think of any good content specifically for DE?	Content doesn't matter much	Hardware is probably the best content in computer science  How to cook, how to make craft, how to do mechanical stuff , calligraphy, etc. – something that shows process  Foreign language/business classes don't show off much	In order to show the power of the DLA, it should show that it can be used in any academic area (humanities, education, biology, etc.)  At least the course index page should have a list of other kinds of courses  Should show a variety – any course, any type of material  Maybe better to have one fully mocked up demo and a few smaller mock-ups for a variety
What other media are good to be used for DE?	Shockwave, Java, etc.	"Whiteboard" technology – allows students to draw pictures and save them. Sometimes it can be drawn live. Called "Net Meeting"? Nice to have live video sessions if everyone is logged on at the same time	Sound files for foreign language education – visual information is not as important
How should students be evaluated? Are multiple choice questions good enough?	Nice to offer options to choose other ways to evaluate   Interactive test using shockwave (dropping off draggable objects)  Should have practical exercises	Tough to think of  Students can cheat with books easily in a DE course  Project based evaluation – do and submit a project on the web	Participation in chat room / news group discussion  Nice to have a chat section as a requirement  Must post news  If everyone is logged on at the same time, teachers draw something with whiteboard – use "emotional cues" (smiley faces/ facial expressions/ bar graphs) to express students’ status (no clue/ fine/ question/ stop sign, etc.)
Other comments?	If there is a purpose, having a different number of frames at a time is OK – more important to show information properly than consistency  Consistency is a big issue  What can a guest user do?	Nice to have a link for technical support  Should have on-line chat room sessions?  Overall it is nice because it is simple For Student mode Should have a warning sign that students actually don't remove CPU – maybe in Help  Class starts on the same day or starts any date that a student wants to start? Print out button – can be tricky to print out a frame based page using the print button in the web browser	The upper right frame is too big for the amount of the information it holds – it is also kind of repeat of the thing in the upper left side  In Tapped-In, there are different ways to ‘emote’  Should have hands-on interaction, not just watching video  New type of exercises such as cabling parts and moving CPU by using mouse