Embedded Files
Spring 2023, Dave Song
<Introduction>
The paper focuses on the usage of VR factory layout planning processes. Objectives of the paper included: evaluating the collaborative capabilities of the system and exploring the limits of the VR system used.
Specifically, factory layout planning requires collaboration among multiple stakeholders and frequent communication among them in order to achieve a finely crafted layout plan. Using the proposed VR system targeting the VR Factory Layout Planning process, the paper examines and evaluates the equality of planning and collaboration.
<Key Research Question>
“What are the affordances and limitations of a multi-user virtual reality system for supporting layout experts, project managers, and project members with expertise in logistics, human factors, and maintenance in their collaborative task of planning and evaluating factory layouts of vehicle manufacturers as measured through usability inspection and collaborative joint attention?”
“it is important to do an extensive user evaluation, in order to make sure the innovation and development contribute to more effective and pleasant collaboration”
<Experiment, Study Design>
For the evaluation, the paper defines a few key elements in its collaborative system
Coding or Annotating: “an analytical process in which data, in both quantitative or qualitative form, is categorized to facilitate further analysis. Ofentimes the process help”
Collaboration: “the action of working with someone to achieve a goal. Exchanging information is essential for collaboration. Explaining an idea is a building block of collaboration.”
Experience: mostly referring to user experience, meaning the overall experience of a person using a product such as a website or a computer application, especially in terms of how easy or pleasing it is to use.
Gaze: the act of seeing and being seen; a steady intent look or stare. In the context of this research, the user's gaze concerns both the position of the head and the field of view (perspective) of the user, i.e. the position of the eyes is not considered.
Joint Attention: the shared focus of two individuals on an object. It is achieved when one individual alerts another to an object by means of eyegazing, pointing or other verbal or non-verbal indications.
Learner: the complementary expert, the listener. the critical opponent to the teacher. (Participant B)
Teacher: the explainer, the layout expert. "the person tha both has to prove a point as well as ask for feedback. (Participant A)
User Evaluation
discusses a few evaluation methods to get qualitative and quantitative data.
NASA TASK Load
subjective
multi-dimensional scale: Estimates from one or more operators
It has reached the status of a de facto standard. Assesses system usability on ten 5 point scales from strongly disagree to strongly agree.
Another usability testing → Heuristic Evaluation.
Observing interactions between user to user user to a computer system
operationalize and analyze observations
<Method>
Goal of the experiment was to simulate real world practice of planning factory layouts in the proposed VR system. The process was analyzed with the comparative portion of the research where participants were asked to do the same collaboration process with 3D representation of the design with desktop monitors.
During the collaborative work process, participants' views were recorded. For evaluation quetionnaire, SUS was used. They also further analyzed the usability with the video analysis. For the comparative portion of the experiment, LayCAD — a factory design software — was used.
There were two main tasks participants performed:
In pairs, participants A “explains the design decisions of the room to participant B”. While explaining the decisions, participant A could measure the distance between objects.
Participants were asked to rearrange objects in VR environment.
<Analysis>
Qualitative Analysis with coding for CJA
the video, audio and screen recordings are combined for the purpose of analysis. They identified collaborative actions into 7 categories for the purpose of the task1. The they measured the total duration of CJA which is defined as a period of time participants communicate ideas and information through verbal and non-verbal methods.SUS Analysis
the average of 69.8 = above average
below 68 is below average.
Comparative analysis of the VR system with LayCAD
Indirect Heuristics
points out that match between the system and the real world is an important heuristic to consider
While some of the features to fit the purpose of the system is necessary
Platform convention → is therefore important to follow
Avatars → evaluation. Overpromise and underdeliver?
Physical environment limits the vr remote collaboration?
Latency as well as glitchiness → should be also measured with a bench mark to come.
Evaluation Questionnaire Used in the study
Evaluation Questionnaire Used in the study
Key Takeaways(for collaborative VR system evaluation):
Combination of Heuristic, SUS, and comparative experiment
Set of advice about ideal VR systems
reflecting real world environment
SUS standards
Some of the characteristics examined in the experiment can be used as key components for the VR collaboration evaluation
Attempts to provide extensive and holistic evaluation of the VR system used in the experiment.
Also points out the fact that no standard for VR collaboration evaluation can make avoidable hinderance in vr collaboration research and higher quality of VR collaboration experience due to lack of comparable metrics that can be applied from a study to another or from a software evaluation to another.
Page updated
Google Sites
Report abuse