Experiment 1: Efficacy of Training using a Head-Mounted Display
James Kelly: Research Student.
Epicentre.
University of Abertay, Dundee.
E-MAIL [email protected]
Abstract
This study was conducted to investigate the usability and efficiency of presenting instructions on a head-mounted display to build abstract models compared to using desktop and paper instructions. There were twenty-one participants in this study, eleven males and ten females who were students at the University of Abertay Dundee. Participants were asked to build three models with play bricks in three conditions, with paper instructions, desktop instructions and with instructions on a head-mounted display. They were timed building the three models in each condition and the errors they made whilst building the models were counted. After the experiment the participants were asked questions in a semi-structured interview to elicit their views of the study. The hypothesis was that there would be a difference in the completion times taken for the models in each condition. The hypothesis was not supported, however feedback from the semi-structured interviews revealed that some participants had perceptual problems following the instructions in the microdisplay condition, whilst others found this instructional system easy to use. These findings are discussed in relation to the way the information is presented in microdisplays.
Introduction
According to Ockermann, Najjar, Thompson, Treanor and Atkinson (1996) the modern workplace is an environment that is in constant flux. In recent years there has been an increase in downsizing and automation in modern advanced capitalist societies. These global changes in production methods require fewer employees operating more complicated automated systems. This constantly changing work environment requires a more flexible workforce that can be retrained in new skills very quickly. This situation often necessitates training that is more efficient than traditional classroom methods. Conventional training methods are viewed as costly and time consuming. The traditional method has been criticized for being removed from the context of the job and there are problems with employees transferring what they are learning to the workplace. Workers trained in the traditional method often forget the training they receive in the classroom when they move back to their work environment. In the view of Carr (1992) cited in Ockermann et al. (1996) traditional training is seen as being geared towards the achievement of classroom goals rather than job performance. Desktop computers can provide more efficient instruction programmes, however workers who maintain automated systems may have to travel to the automation, so would require a more mobile training scheme.
Many researchers in education and instructional technology see the wearable computer as a panacea to many or all of the problems of traditional retraining. As defined by Mann (1996) a wearable computer is a portable display, which can be worn unobtrusively by the user, thus freeing the hands for other activities. These displays are head mounted and some permit the viewer to see the real world as well as information on the display. It is claimed that wearable computers can train workers in task specific skills such as a psychomotor skill or a learning a specific computer package. Conditions such as myopia, hyperopia and astigmatism are common in the population. Using head-mounted displays could cause problems for people who are precluded from wearing their normal eye correction. There is also the situation where the person overestimates the effectiveness of his or her eyesight and requires stronger eye correction. An important issue arises when training older workers whose eyesight naturally declines with age. This syndrome which comes under the rubric of presbyopia which is a refractive condition where declining accommodation power of the eye makes near vision difficult. Many manufacturers of these headsets tend not to leave enough room for spectacles which is an obvious disadvantage to users who do not have 20/20 vision.
Apart from individual differences in eyesight there is also great individual differences in cognitive style. Rider & Raynor, (1988) postulate that people have different ways of learning information, they usually fall into four general categories; analytic-verbaliser, analytic-imager, wholist-verbaliser and wholist-imager. These groupings reflect the different ways individuals process information, whether they prefer pictures to text and take in information in parts or as a whole. Information should be presented in a way that allows for these differences in cognitive styles. One possible solution to this is to present information on a head-mounted display in different modalities, pictures, text and sound. Najjar (1998) reviewed empirical evidence that information presented in a multi-modal fashion enhances learning. Nugent (1982) found a significant difference in learning when visual and textual representations of the material were presented simultaneously. The programme used in the head-mounted and desktop conditions in this study followed this rule with text and pictures used in tandem in the instructions. By using animation, this study is a partial replication of work by Spangenberg (1973) who found that motion enhanced training in an assembly task.
Stone (2000) describes the important role attention has in learning and how the wearable computer focuses the learner's attention wholly on the task. This raises questions about divided attention or when the user has to switch attention between information projected on a screen in the viewer's near vision and the real world. As Wickens (1992) points out there is empirical evidence that spatial proximity or nearness in space will permit parallel processing or switching attention between tasks. However this type of system does not necessarily guarantee parallel processing. It may seem obvious that instructions for a task superimposed on or visually close to the physical execution of the task itself would facilitate divided attention or parallel processing between channels. According to Wickens (1992) studies in this research area indicate that the segregation of the tasks may be characterized in terms other than visual location. They may be defined in terms of the nature of the task or even the assumed distance from the observer.
This study investigated whether there are any differences between the efficiency and usability of a wearable computer compared to traditional methods. Thirty participants were asked to build three different abstract models from play bricks in three conditions. In one condition the instructions were simple paper instructions. The other two conditions had the paper instructions converted into an interactive computer programme. In one of these conditions the instructions were presented on a desktop, on the other condition they were presented on a head mounted microdisplay. The participants were timed building the three models and the number of errors they made performing this task was calculated. After the experiment the participants were asked about their views on the experiment using a semi-structured interview. The experimental hypothesis is that there will be a difference in the completion times for the models in each condition.
Method
Design
This pilot study was in two parts, a quantitative section and a qualitative one. The quantitative section employed a within subjects design. The independent variable was the instruction format used to build the models. The dependent variables were a) the time taken to construct the models in seconds and b) the errors made whilst building the model. In this study the criterion for an error was the placing of a brick on a model in the wrong place or in the wrong orientation. If a participant placed a brick wrongly but realized their mistake and corrected their error immediately, this was not counted as an error. The qualitative section involved a semi-structured interview with set questions to obtain feedback from the participants about the experiment.
Participants
There were twenty-one participants, eleven males and ten females with an age range from eighteen to forty-five. The participants were students of the University of Abertay Dundee. All the participants were volunteers. Participants were selected using convenience sampling. All the participants had normal or corrected eyesight. Originally thirty participants were tested. However the data from nine were omitted from analysis because they did not wear their eye correction in the head-mounted display condition. These participants did not normally wear their glasses for viewing a computer so did not wear them to look at the projected image in the head mounted display condition. The image in the head-mounted display is projected at about arm's length from the viewer, so perhaps outside the range of the viewer's visual acuity. The results from these participants may have been confounded by their inability to follow the instructions in the head-mounted display condition.
Materials and Apparatus
Materials in this study comprised of two computers, a Sony Glasstron binocular LCD head-mounted display that could view output from PCs (see appendix 1). The Glasstron headset allows the user to wear their glasses while viewing the projected display. Play bricks to construct three models (see appendix 2). The models in the study were referred to as "Alpha", "Gamma" and "Delta". Each model had 27 bricks and had similar characteristics in construction. There was a similar angle of view of each model in the pictorial instructions, type of bricks and complementary instructions. The paper instructions (see appendix 3) consisted of a booklet with the twenty-seven pictures of the stages required to build the model. An interactive programme using the same instructions as the paper guide was used in the desktop and microdisplay conditions (see appendix 4). This comprised of a screen presentation, with an animation of the model on the right hand side and a stage controls on the left. Participants could go to any stage in the animation or move through the assembly frame by frame. Video cameras were used to film the participant's hands whilst constructing the models and the completion times were calculated in seconds using video recording machines.
Procedure
The order in which participants were assigned to the conditions was counterbalanced (see appendix 5). The three display types in which the instructions were presented to the participant were paper, desktop and head-mounted display. The same instruction programme was used in both the desktop and microdisplay conditions. One PC was used for the desktop condition and the other for the microdisplay condition. Before using the desktop and microdisplay instructions, the experimenter carefully explained the control system to the participant. The participant was asked to sit in one of three testing bays. One bay contained the head-mounted display condition, one the paper and the other the desktop condition. In the latter condition the participant's head was positioned fifty-five millimeters from the screen to give participants the same field of view, as they would encounter in the head-mounted display condition. In the head-mounted display condition the participant was asked to wear their eye correction, if they normally did so to look at a computer screen. The participant was asked to construct a different model out of play bricks in the three conditions, this was done to counteract practice effects. The participants were also asked to build the model in a predefined area to ensure that their hands would be captured by the video cameras. After the participant had completed the test battery, the experimenter asked a set of questions about the experiment and noted the responses on a feedback questionnaire. The questionnaire took a semi-structured format and comprised of seven questions that try to elicit feedback about the mechanics of the programme and about the three instruction formats (see appendix 6). The completion times and errors were calculated during video play black.
Results
Figure 1. Graph showing mean completion times and standard error for each instruction condition: Paper, Desktop and Head-Mounted Display.
Figure 1 illustrates the mean completion times and plus or minus 1 standard error for the three models in each instruction condition. The paper instruction condition has the highest mean completion time of 166.2 seconds. The head-mounted display condition has the second highest mean completion time of 154.5. The desktop instruction condition has the lowest mean completion time of 150.2. The standard error for the paper and desktop are 9.5 and 8.6 seconds respectively. The standard error for the head-mounted display condition is 11.2 seconds. To test these differences further, the data were analysed using a one-way analysis of variance for repeated measures. There was no significant effect for type of instruction F (2,40)=0.962,N.S
Figure 2 shows the mean number of errors made in each of the three conditions. The paper condition had the lowest mean number of errors with 0.86. The desktop and head-mounted display had similar mean errors with 2.0 and 2.1 mean errors respectively. To test these differences further, the data were analysed using a one-way analysis of variance for repeated measures. There was no significant effect for type of instruction F (2,18)=0.232,N.S
Figure 2: Graph showing total errors made in the three conditions.
Figure 3: Graph showing the frequency of errors made in the three conditions.
As illustrated by figure 3 the frequency of the number of times an error was made was different for each condition. In the paper instruction condition participants made either one, two four or no errors whilst building the model. In the desktop condition participants made a wider range of errors. They either made one, two, three, five, ten or no errors. Participants made the widest range of errors in the head-mounted display condition. They made either one, two, three, four, six, seven, eight or nine errors.
Feedback Questionnaires
This study also had a quantitative section that involved a semi-structured interview with each participant immediately after he or she had completed the test battery. The questionnaire had seven questions about the study, about the mechanics of the interactive programme and the instruction formats in the three experimental conditions. The first question asked which of the models the participants found the most difficult to build. This was to ascertain whether the models did have similar characteristics and that no model was harder to construct than the others. Of the twenty-one participants, four found the Gamma model the hardest, three found the Beta the hardest, but fourteen thought that the three models had the same level of difficulty. To test for a significant difference between the observed frequencies a chi square test was carried out on the data and the result showed that a significant number thought the models were the same, x2= 7.714, p< 0.05.
In question two the participants were asked which instruction format they found the easiest to use, ten said paper instructions, four said desktop and seven said head-mounted display. A chi square test was carried out test if there was any significant difference in instruction preference. The result showed no significant difference, x2= 2.571, p>0.05. This question had a second part where the participants were asked which instruction format they found the hardest to use. Five participants found the paper instructions the hardest, seven thought the desktop was the hardest and nine thought the head-mounted display condition was the hardest. To test for any significant difference in instruction dislikes the most a chi square test was carried out. The result shows no significant difference, x2=1.143, p>0.05.
Question three was another question that attempted to investigate participants' views on the mechanics of the programme, namely the speed of the animation. The speed of the animation was one frame every 4.5 seconds. Ten participants thought that this was too fast, three participants thought the speed was too slow and eight thought it was just about right. To test whether there was a significant difference of opinion on speed, a chi square test was carried out and the result showed no significant difference, c2=3.714, p>0.05. Question four asked participants whether they found the control system in the interactive programme easy to use. All participants except one thought that the control system was easy to use. Question five asked the subjects whether they had problems identifying the size of the bricks in any of the conditions in the study. Seven said they had difficulty in identifying the size of the bricks. Two of those participants remarked that they did not bother to read the textual information on the brick, e.g. 2x4 for two by four and picked up the wrong brick. On the other hand fourteen participants had no difficulty in identifying the size of the bricks. One of this particular group of participants felt that this textual information helped them to choose the correct brick. To test whether these observed frequencies were significant a chi square test was undertaken on the data. The result was that there was no significant effect for identifying the bricks, x2=2.333, p>0.05.
In question six the participants were asked whether or not they wore eye correction. Four participants wore eye correction all the time the remainder never wore eye correction. The last question asked specifically about using the head-mounted display for building the models. Here many participants gravitated to one of two positions regarding the proposed advantage of having to just move the eyes to switch attention between the construction area and the head-mounted display to do the task. Six participants found this switching attention very difficult. Some found they lost their place following the instructions looking up and down, they experienced difficulty with coordination and could not take in the instructions fast enough. Some felt that switching their attention between the head-mounted display and the task slowed them down. One participant commented that is was easier for them to move their head rather than their eyes. Another person in this group felt that their attention on the head-mounted display was split in two, between the image of the brick and the animation. On the other hand eight participants liked the feature of discretely moving the eyes between the screens in the head-mounted display and the task. Conversely they found an advantage in not having to move their head and one felt that this system speeded up their performance in constructing the model. Two participants had no further comments to make about the head-mounted display. One participant thought the projected image on the head-mounted display looked far away. Another thought that more control of the programme was needed for the head-mounted display condition especially a pause function.
Discussion
Results from this study do not support the hypothesis that there would be a difference in mean completion times for the different types of instruction formats. The desktop condition had the fastest mean completion time but was not significantly faster. The head-mounted display had overall the highest error rate compared to the other two conditions. The paper condition had the lowest rate of errors. The lower number of errors in the paper condition coupled with the slower mean completion times is perhaps indicative of a speed-accuracy trade off. In this condition the participants were working at their own pace and not at the pace the computer programme that presented the instructions in the other two conditions. There was a wider spread of errors made by participants in the head-mounted display condition whereas in the desktop and paper conditions nearly all the participants made no more than five errors.
The feedback from the semi-structured interviews reveals that most of participants preferred the paper instructions. The instructions on the head-mounted display were the least popular. Opinions on the physical act of using the head-mounted display to do the task largely polarized the participants into two groups. Firstly those who felt that the head-mounted display was easy to use and found the switching between the instructions and the task an advantage using just eye movement. Secondly, those who had difficulties with switching their eyes back and forward in this manner. This supports Wickens (1992) who found that although close proximity in space of one task might allow divided attention or switching attention it may also cause confusion between similar tasks. Two activities demand more cognitive resources than one. There is also the question of task difficulty. If one task is perceived as more difficult than the other it will receive more attention. Some participants found following the instructions in the head-mounted display difficult and this interfered with executing the task. Seven participants had difficulty identifying the size of brick they had to choose from the instructions, despite the information presented in text and pictorially. Some did not pay attention to the text on the bricks but chose the bricks according to their shapes. This would support the idea that people take in information differently. Some may find the multi-modal presentation useful others may not.
The feedback questionnaires also revealed important information about the instruction programme and the materials used in the experiment. A significant number of participants thought that the models had similar characteristics so no one was model was harder than the others. This meant that the participants were building a similar model in each condition and this counteracted practice effects. Ten participants thought the speed of the interactive programme was too fast. This supports a previous study by McGuire (1961) cited in Najjar (1995) that showed that learners perform better with slow motion instructions. All the participants apart from one thought the control system in the interactive programme easy to use.
A future study would take note of the quantitative and qualitative results of the present study. Participants will have increase control over the instruction programme allowing them to pause it at will and have a better sense of the place they are at in the model construction. The conditions will be changed and a second type of head mounted display will replace the paper instruction condition. The latter head-mounted display allows the user to make shorter eye movements whilst switching attention from the instructions to the task. This would test whether it is advantageous to have a shorter attention switching distance. The instructions will be a video file of a pair of real hands performing the task rather than an animation of the stages of construction. There will be an attempt to make the instructional format as explicit as possible. The experimenter will ensure that all participants have corrected vision whilst using the head mounted displays.
Postscript
As mentioned in the participants section data from nine of the original thirty participants were discarded. Participants were asked to wear their eye correction before using the head-mounted display. These nine participants did not normally wear eye correction for viewing a computer screen so neglected to wear eye correction for the head-mounted display condition. The performance of these participants may have been compromised since they may have needed their glasses for distance. This group was following the instructions on a projected image, which was possibly outside the range of their visual acuity. This initial oversight may serve as a caveat to any experiments with head-mounted displays. In this type of experiment it is crucial to ensure that all the participants have corrected vision.
References
Carr, C. (1996) in Ockerman, J.J.,Najjar,L.J., Thompson, J.C., Treanor, C.J. & Atkinson, F.D. FAST:A Research Programme for Educational Performance Support Systems. http://mime1.marc.gatech.edu/mime/papers/edmedia97_demo.html
Ockerman, J.J.,Najjar,L.J., Thompson, J.C., Treanor, C.J. & Atkinson, F.D. (1996) FAST:A Research Programme for Educational Performance Support Systems. http://mime1.marc.gatech.edu/mime/papers/edmedia97_demo.html
Mann, S. (1996) Wearable, Tetherless, Computer-Mediated Reality
(with possible future applications to the disabled) http://hwr.nici.kun.nl/pen-computing/wearables/steve-mann/
McGuire W.J. (1961) in Najjar,L. (1995) A Review of the Fundamental Effects of Information Presentation in Multimedia Learning. Georgia Institute of Techology. Technical Report GIT-GVU-95-20
Najjar, L. (1988) Principles of educational multimedia user interface design. Human Factors 40, 311-323.
Nugent, G.C. (1982) Pictures, Audio and Print: Symbolic Representation and Effect on Learning. Educational communication and Technology Journal. 30. 163-174.
Riding, R. & Rayner, S. (1998) Cognitive Styles and Learning Strategies: Understanding Style Differences in Learning and Behaviour. David Fulton.
Spangenberg, R. W. (1973) The motion Variable In Procedural Learning. A.V. Communication Review, 21, 419-436
Stone, R. (2000) in Karkowski, W. (ED.) International Encyclopedia of Ergonomics and Learning Styles. Taylor and Francis.
Wickens, C.D. (1992) Engineering Psychology and Human Performance. Second Edition. Harper-Collins.
Appendices
- Appendix 1�������������Sony Glasstron Head-Mounted Display.
- Appendix 2�������������Abstract Models
- Appendix 3�������������Paper Instructions.
- Appendix 4�������������Desktop and Micro Instruction Programme
- Appendix 5�������������.Counterbalancing Schedule
- Appendix 6�������������.Post Test Semi-Structured Interview.
Sony Glasstron Head-Mounted Display.
Model Alpha
Model Beta
Model Gamma
Paper Instructions.
Interactive Programme.
These instructions comprised of a set of unpublished web pages created using Liquid FX v 3.1 HTML editor. The stages of construction of the models were made using Lego Creator. The model was constructed in the program. Then each stage was saved as a jpeg file using the screen capture facility of Paint Shop Pro v.6. The jpeg files were then turned into animations. The animation was made using Animation Shop v.2. that ran at a speed of one frame every 4.5 seconds. The right hand panel contained the instructions in a frame by frame format. The left-hand panel contained links to each stage in the animation. The current piece to be used is in the left hang corner of each panel, with an arrow indicating its position on the model. The Start/Restart Animation" button starts the animation from the beginning.
Counterbalancing
Subject 1 || Alpha Paper || Beta Desktop || Gamma Micro
Subject 2 || Gamma Paper || Alpha Desktop || Beta Micro
Subject 3 || Beta Paper || Gamma Desktop || Alpha Micro
Subject 4 || Alpha Paper || Gamma Desktop || Beta Micro
Subject 5 || Beta Paper || Alpha Desktop || Gamma Micro
Subject 6 || Gamma Paper || Beta Desktop || Alpha Micr
Subject 7 || Alpha Desktop || Beta Micro || Gamma Paper
Subject 8 || Gamma Desktop || Alpha Micro Beta Paper
Subject 9 || Beta Desktop || Gamma Micro || Alpha Paper
Subject 10 || Alpha Desktop || Gamma Micro || Beta Paper
Subject 11 || Beta Desktop || Alpha Micro || Gamma Paper
Subject 12 || Gamma Desktop || Beta Micro || Alpha Paper
Subject 13 || Alpha Micro || Beta Paper || Gamma Desktop
Subject 14 || Gamma Micro || Alpha Paper || Beta Desktop
Subject 15 || Beta Micro || Gamma Paper || Alpha Desktop
Subject 16 || Alpha Micro || Gamma Paper || Beta Desktop
Subject 17 || Beta Micro || Alpha Paper || Gamma Desktop
Subject 18 || Gamma Micro || Beta Paper || Alpha Desktop
Subject 19 || Alpha Paper || Beta Desktop || Gamma Micro
Subject 20 || Gamma Paper || Alpha Desktop || Beta Micro
Subject 21 || Beta Paper || Gamma Desktop || Alpha Micro
Subject 22 || Alpha Paper || Gamma Desktop || Beta Micro
Subject 23 || Beta Paper || Alpha Desktop || Gamma Micro
Subject 24 || Gamma Paper || Beta Desktop || Alpha Micro
Subject 25 || Gamma Desktop || Beta Paper || Alpha Micro
Subject 26 || Alpha Desktop || Beta Paper || Gamma Micro
Subject 27 || Gamma Desktop || Alpha Paper || Beta Micro
Subject 28 || Beta Desktop || Gamma Paper || Alpha Micro
Subject 29 || Alpha Desktop || Gamma Paper || Beta Micro
Subject 30 || Beta Desktop || Alpha Paper || Gamma Micro
Post Test Interview
Subject :
M or F
1) Which figure did you find hardest to build?
2) Which of the instruction formats, paper, desktop or microdisplay did you find easiest/hardest to use?
3) In the desktop and microdisplay conditions did you find the speed of the animation either too slow or just right?
4) In the desktop and microdisplay conditions did you find the control
system easy to use?
5) Did you have any difficulty in identifying the size of the bricks.
6) Do you wear glasses or contact lenses.
7) Do you have any further comments about the test? For Example the
head-mounted display condition?