Choosing Appropriate metrics to compare

Overview:

In this section, we explore all the different metrics you can use to create a comprehensive wiki entry on a data visualization software for your project. Before we choose which metrics to focus on, we must revisit the purpose of the wiki entries we’re creating, as well as the wiki as a whole. Whether it be about a tutorial on how to develop a specific plugin, a comparison of different collaborative VR software, or a page detailing a specific VR headset’s tech specs, our roles as students and wiki contributors are to share information about VR to the general public. However, the viewers of this wiki use the information on wiki pages to learn more about and compare different software, hardware, plugins, or tutorials with each other. This is why it is our job as wiki contributors to make each wiki page as detailed, unique, and comparable as possible. 

The Metrics:

Following the ideas outlined in the seven scenarios paper, we combine some ideas and split the possible metrics into four categories as shown below. 

Usable Environments

We try to figure out who exactly the software is meant for, in what industries it is applicable in, and at what point in the workflow the software is useful.

• Intended Audience

  • This section should qualitatively describe who the potential users of the software might be. Keep in mind the learning curve to use the software, as well as the key features that could make the software specialized for a particular industry. 

• System/Hardware Requirements

  • This section should describe the minimum and recommended system requirements (for PC and/or VR hardware), as well as VR hardware compatibility.

• Databases/Usable File Formats

  • This section should list the online databases it can connect to, as well as the import and export file types it supports. Note that some software have different database/file format compatibility depending on the type of license you use. Include a link to a tutorial on how to load data, or make a step by step tutorial yourself. 

• Prior Industry Knowledge Requirements

  • This section should give a general insight into how much prior knowledge is assumed in order to use the software. Try to split up the analysis into three parts: complete beginner, some experience, expert in the field, and list the expected difficulty for each demographic. The difficulty can be described in a multitude of ways: 

    • Subjective opinion on a scale of 1 to 10

    • Time it takes to use a certain key feature in the software

    • Time it takes to setup the app

    • Level of jargon the software uses

• GUI

  • This section should state whether or not a GUI exists for the software, and include a gallery of the GUI or CLI.

• Customizable Plugins (if applicable)

  • This section should state whether or not the software supports customizable plugins. List a few example features that could be implemented by creating customizable plugins. Link a related wiki tutorial on customizable plugins if one exists.

• Cost

  • This section should list all different tiers of paid content, as well as the features each tier contains. 

• Integration of third party applications/services (if applicable)

  • This section should list all different third party applications/services that are integratable with the software.

• APIs (if applicable)

  • This section should list all APIs the user can utilize. 

Communication and Collaboration

We try to figure out how suitable this software is in communicating ideas through data visualization, as well as the collaboration features within the app. 

• Collaboration room size

  • This section should list the minimum and maximum room sizes for collaborative VR spaces. Note that a “maximum” room size doesn’t mean that the software is usable/smooth at that capacity, so try to test this out with an in class activity, or research any user experiences online. 

• Private/Public Collaboration Rooms

  • This section should describe the privacy of collaborative VR spaces. Note that some software have different privacy settings depending on the type or tier of license. 

• Methods of Communication

  • This section should describe the methods where users can communicate through a verbal, visual, and tactile medium. Some examples of this would be an in game voice channel, avatars with movement, laser pointers, etc.  

• Reliability of Communication Features

  • This section should test how usable and reliable the methods of communication are. Usability and Reliability can be measured in a multitude of ways:

    • The Replacement Method - Ask yourself whether the in game communication feature can be replaced by a “better” external communication tool. (ex. in-game audio chat with zoom in the background) 

    • Testing Latency - How much of a lag (in terms of time) is there when using a specific communication feature?

    • Testing Consistency - Does the method of communication consistently work/render? How many times is the functionality buggy/unavailable in a 15 minute session? Note that network connection may heavily affect the consistency of communication functionality, so make sure to take that into account. 

• Methods of Collaboration

  • This section should describe the methods in which users can collaborate through a verbal, visual, and tactile medium. Some examples of this would be a shared whiteboard, the ability to modify the same data model, other features that more than one participants have shared access to.  

• Reliability of Collaboration Features

  • This section should test how usable and reliable the methods of collaboration are. Usability and Reliability can be measured in a multitude of ways:

    • Testing concurrency - Can a feature be used by more than one user at the same time (ex manipulating data models)? List any bugs/glitches that occur when trying to use the same feature simultaneously.

    • Testing latency - How much of a lag (in terms of time) is there when using a specific collaboration feature?

    • Testing Consistency - Does the method of collaboration consistently work/render? How many times is the functionality buggy/unavailable in a 15 minute session? Note that network connection may heavily affect the consistency of communication functionality, so make sure to take that into account

• Avatars (if applicable)

  • This section should record the functionality and appearance of the avatars. Some feature to look out for that are specific to avatars include:

    • Customizability

    • Realness (Whether it is based off your own face or not)

    • Displays hand motions

    • Mouth movements when speaking

  • Check out a google doc on Avatars that Shenandoah made for more info on how to evaluate avatars.

Software Performance

We try to figure out how efficient the software is in completing specific tasks. An example of this would be the time and difficulty of setting up the app in order to be able to start to use its core features.

• Visualizable Dimensions

  • This section should note down what kind of data it is displaying, as well as the maximum number of dimensions it can display. 

• Features

  • This section should list out any features that the software has. Each feature in the list should include what license levels can access it, how long it takes to use that feature, what prior knowledge is required to use that feature, and bugs that occur when using the feature.

User Experience

We try to figure out what efforts are made in order to improve the user experience of the app, as well as how successful each effort is in making the app more user friendly.

• Movement

  • This section should describe how movement is handled in the software. Below is a list of questions you should consider when describing the movement. Note that various software often have the option to switch between different ways to move.

    • How can you “look around”? Do you need to use the handheld controller, or can you move your head?

    • Does the software require a non-stationary environment?

    • What actions do you need to take to move within the software? (ex. Move the joystick, point and click, etc.)

    • What does movement look like within the headset? (ex. Teleportation, continuous movement, etc.)

  • Check out Michael's wiki entry on user movement for more info,

• User Controls

  • This section should describe the controls of the software. The key user controls should be listed, or a link to a documentation describing the key controls.

    • Menu

    • Movement controls

    • Zooming

    • Panning

    • Grabbing

• Intuitive User Experience

Resources

We try to cover information we weren’t able to get to in the categories above (ex. documentation, user support, tutorials)

• Official Installation Tutorials

  • This section should provide a link to the official installation guide on the software website. If it doesn’t exist, please try to make a video tutorial/step by step guide.

• User Support

  • This section should provide a link/details for user support.

• Official Documentation

  • This section should provide a link to the official documentation for the software.

• In Game Tutorial

  • This section should denote whether or not the software has an in-game tutorial to get acclimated with the controls.

• Tutorials

  • This section should list out all related tutorials that exist within the wiki.