PROPOSALS

Project 1 Proposal 

Presentation for Project 1 Proposal

In-Class Activity: Mozilla Hubs

End Presentation for Project 1 

Project 2 Proposal 

Presentation for Project 2 Proposal

Poster 

In-class Activity:  Collaborative 3D Modeling in VR 

Public Demo 

ALL OTHER WIKI CONTRIBUTIONS

CONTRIBUTION 1: MeshLab Tutorial for Cleaning 3D Models - Here

CONTRIBUTION 2: 3D Slicer Tutorial on generating 3D Model from Biomedical Imaging - Here

CONTRIBUTION 3: Wiki Page for Mozilla Spoke - Here

CONTRIBUTION 4: Wiki Page for Unity ProBuilder - Here

CONTRIBUTION 5: Wiki Page for Unity ProGrids - Here

CONTRIBUTION 6: Wiki Page for Unity Polybrush - Here

CONTRIBUTION 7: Mozilla Hubs In-Class Activity - Here

CONTRIBUTION 8: Blender XR Wiki Page - Here

CONTRIBUTION 9: Slicer VR Wiki Page - Here

CONTRIBUTION 10: Maya MARUI Wiki Page - Here

CONTRIBUTION 11: Collaborative 3D Modeling in VR In-Class Activity - Here

CONTRIBUTION 12: Oculus Quest 2 Hand Tracking Setup Tutorial - Here

CONTRIBUTION 13: Working with MRTK In Unity - Here

1/27/21 - 5 Hours

• Looking into 3D software packages and how they can be brought into VR. Rated apparent difficulty of porting to VR.

  • Blender (Hard)

    • Has VR integration in v2.83, so this is worth looking into

    • No collaboration

  • Maya (Moderate-Hard?)

    • Industry-standard

    • No built-in VR support 

      • VR plugin exists: https://vr-plugin.com/

      • Collaborative plugin created by community member

  • Protospace (https://www.youtube.com/watch?v=dD0FoH8M1EM) (Not feasible, but interesting)

    • Doesn't seem to be commercially available

    • Cool concept (Uses HoloLens AR for collaboration)

    • Overlay models on real-world objects

  • Unity (Moderate)

    • Built-in VR packages

    • Collaboration via Unity Teams, but not realtime

    • Realtime multiplayer supported (Photon), so perhaps the experience can be built in-game rather than in-editor

  • 3D Slicer (Hard)

    • Biomedical CT segmentation software (I used this extensively in the Brainerd Biomechanics Lab at Brown)

    • Allow to both visualize and segment scans of organisms

    • VR space may allow for more clarity and increased understanding of underlying structures

• Played around in virtual spaces with another classmate to test realtime VR interactions and visualization possibilities

  • RecRoom

  • VRChat

  • Mozilla Hubs

• Project Ideas:

  • Viewing and editing 3D models /FBX files collaboratively in real-time using Unity

  • Use Maya plugin to allow for the collaborative creation of animation and 3D modeling/sculpting

  • Develop multi-user data visualization interface in Unity: highlight, manipulate, compare, and explore data in new ways

    • Point-clouds, multi-dimensional graphs/charts, interactive data visualizations

  • Realtime multi-user editing of biomedical imagery using 3D Slicer

1/28/21 - 2 Hours

• Looking into multiplayer systems in Unity (Photon) and VR in Blender 2.83 to see what's feasible for a project

• Reviewing some 3D software from my previous lab's wiki: https://gitbook.brown.edu/xromm/model-generation 

• Added link to the wiki page I wrote for MeshLab: https://gitbook.brown.edu/xromm/model-generation/xromm-cleaning-3d-models-with-meshlab 

  • Could also add link to page I wrote for Slicer, but unsure if relevant (https://gitbook.brown.edu/xromm/model-generation/xromm-creating-3d-models-with-3d-slicer)

1/31/21 - 4 Hours

• Looked through previous documentation in wiki 

• Assessing existing technologies for 3D modeling in VR:

  • BlenderXR

    • Open source

    • Supports Blender Editing in VR, so OBJ, FBX, DAE, CAD, etc. files can be manipulated in realtime

    • Still in beta

    • Seems to be single-user, so no built-in collaboration 

  • MARUI

    • Not open source ($550 lifetime license)

    • Seems to have same features as above but for Maya instead 

    • Developed by same company

  • Mozilla Hubs

    • Open source

    • Runs in-browser 

    • Supports multiple users

    • Supports screen sharing, video calling, 3D object placement, drawing, uploading 3D models and animations 

    • Supports cross-platform collaboration (some users in VR and some on desktop, may even run on mobile using the Mozilla XRViewer app)

    • Contains a scene editor called Spoke (https://hubs.mozilla.com/docs/intro-spoke.html) 

      • Ran it in-browser on Oculus browser and it worked well

      • Supports uploading local 3D model files

      • Doesn't seem to support model editing, only scaling, rotation, and placement

  • Unity ProBuilder

    • Allows for modeling in Unity editor

    • Doesn't translate over to play mode once scene is run

    • Not collaborative 

2/01/21 - 4 Hours

• Tested out Unity scene on Oculus Quest 2

  • A scene could be built in Unity that allows for multiplayer using Photon, with integrated model building and editing

    • Would require a lot of backend however, as this would be building a collaborative experience bottom-up

• Discovered that there's a VR extension for 3D Slicer (Really exciting bc it integrates VR and scientific visualization inherently)

• Potential Projects:

  • Editing Scenes and 3D Models Collaboratively in Blender

    • 3 Things I will do

      • Get BlenderXR working properly on Oculus

      • Explore collaborative potential 

      • Document BlenderXR

    • Activity: Getting everyone to set up and try out BlenderXR would be pretty cool and informative 

    • Deliverables

      • BlenderXR wiki page

      • Workflow for collaboration in BlenderXR 

  • Exploring VR Collaboration in Maya using MARUI - Decided not to pursue due to the steep software price. Software is also essentially the same as BlenderXR (**Has 7 day trial, so may use that and document everything I find in that one week)

  • Exploring VR Collaboration in Mozilla Hubs

    • 3 things I would do

      • Explore the collaborative possibilities of Spoke (does collaboration only have to be limited to scene viewing vs editing?)

      • Test how capable Hubs is for importing data/visualizations

      • Document Mozilla Hubs

    • Activity: Getting everyone in the same Mozilla Hubs instance to test out the functionality and stress test the software

    • Deliverables

      • Mozilla Hubs wiki page

      • Workflow for work using Mozilla Hubs and Spoke

  • Editing Scenes and 3D Models Collaboratively with Unity ProBuilder

    • Things I would do

      • Explore the collaborative possibilities of Unity with Photon (problem with probuilder is that it's in-editor only)

      • Document workflow for Unity Probuilder

    • Activity: maybe walk everyone through getting Unity working with VR? Depends how ProBuilder reacts to VR. Because it seems like a plugin that doesn't support VR or collaboration, it might not be very relevant.

    • Deliverables

      • Documentation

      • Unity Photon and/or ProBuilder setup tutorial

  • Viewing and Editing Meshes in 3D Slicer VR

    • Things I would do

      • Document process for setup, key interactions, and any pitfalls

      • Explore possibility to allow multiple users to view same mesh

      • Write tutorials for getting Slicer to work in VR

      • Ideate on use cases 

    • Activity: maybe walk everyone through getting Unity working with VR? Depends how ProBuilder reacts to VR. Because it seems like a plugin that doesn't support VR or collaboration, it might not be very relevant.

    • Deliverables

      • Detailed tutorial on workflow for getting 3D mesh in Slicer 

      • Documentation on getting VR up and running in Slicer

      • Documentation for any issues and or areas for improvement 

2/03/21 - 3 Hours

• Watched more software demos and read up on documentation

• Decided on project: assessing 4 software packages listed above and trying to expand on them

• This project will be divided into two phases

  • Phase 1

    • Install, explore, and document all use cases for the software (BlenderXR, Unity ProBuilder, Slicer, Mozilla Spoke/Hubs)

    • Find potential pitfalls and ways to expand

    • Create tutorials, document metrics and features

    • Evaluate based on usability, collaboration, effectiveness, scientific visualization

  • Phase 2

    • Based on evaluation, consider possible improvements 

    • Ideate on a software that cold include these improvements 

• Thinking phase 2 can be further elaborated for project 2

• Project 1 Milestones

  • 2/9 - Install, assess, and document Mozilla Spoke/Hubs

  • 2/11 - Continue assessing and documenting Mozilla Spoke/Hubs, start with ProBuilder

  • 2/18 - Continue assessing and documenting ProBuilder + BlenderXR

  • 2/23  - Continue BlenderXR + ProBuilder or MARUI for Maya

  • 2/25 - Continue ProBuilder or MARUI  + Slicer (add to existing Slicer page)

  • 3/2 - Have wikis completed. Polish wiki pages and tutorials + start draft on proposal for expansion and prototyping (?)

  • 3/4 - Prototyping for a new potential software to include functionality not already present (or lacking) in these packages. Setup for potential implementation in project 2. If one of the software packages has particular potential, ideate on how to expand it to be more collaborative

  • NOTE: Blender is set to expand its native VR capabilities in 2021. If this happens before March, that may be another good direction to explore

2/06/21 - 3 Hours

• Used Mozilla Spoke + Hubs, tested collaboration with George

• Tested browser + vr collaboration

• Took notes for wiki 

2/08/21 - 4 Hours

• Wrote tutorial in wiki for Meshlab workflow: Meshlab: Cleaning 3D Meshes 

• Wrote tutorial in wiki for 3D Slicer workflow: 3D Slicer: Generating Scientific 3D Models 

• Started page for Mozilla Spoke

• Evaluated project 1 plan:

  • The proposed project clearly identifies deliverable additions to our VR Software Wiki (5)

  • The proposed project involves collaboration in VR (5)

  • The proposed project involves large scientific data visualization along the lines of the "Data Types" wiki page and identifies the specific data type and software that it will use (3)

  • The proposed project has a realistic schedule with explicit and measurable milestones at least each week and mostly every class (5)

  • The proposed project includes an in-class activity (4)

  • The proposed project has resources available with sufficient documentation (5)

• Created powerpoint presentation for project

  • Software: Mozilla Spoke, Unity ProBuilder, BlenderXR, Maya MARUI, Slicer

2/10/21 - 4 Hours

• Made and tested a scene in Mozilla Spoke

• Wrote Spoke wiki page

• Made Spoke tutorial

2/11/21 - 2 Hours

• Reformatted journal elements

• Got all volume rendering demos working 

• Started planning in class activity

2/16/21 - 2 Hours

• Set up and evaluated Unity ProBuilder, ProGrids, and Polybrush

• Started ProBuilder wiki page, planning to include info on ProGrids and Polybrush, or make separate pages for each

Evaluated journal:

Journal activities are explicitly and clearly related to course deliverables [4]

Deliverables are described and attributed in wiki  [4]

Report states total amount of time  [4]

Total time is appropriate  [5]

Journal Evaluation by George:

Journal activities are explicitly and clearly related to course deliverables [4]

Deliverables are described and attributed in wiki [4]

Report states total amount of time [4]

Total time is appropriate [4]

Evaluated Kota's Journal and George's Journal (there was a mix-up so I ended up doing both... the more the merrier?)

2/17/21 -  4 Hours

• Made pptx showcasing project progress

• Made wikis and set up tutorials for Unity ProBuilder, ProGrids, and Polybrush

3/10/21 - 3 Hours

• Looking more at MRTK, Oculus integration for Unity, Photon multiplayer plugins

• Added to final presentation

3/12/21 -  4 Hours

• Brainstorming project 2

• Evaluating criteria of project 1 software

• Playing around more with Unity plugins 

• Ideating on best ways to make 3d modeling demo 

3/14/21 - 4 Hours

• More Unity tests and ideation, drafting project 2

3/15/21 - 2 Hours

• Drafting project 2

Project 2 Draft:

• • • Idea

      • Collaborative 3D Modeling in Unity with hand tracking. I haven't seen this done, and would be interested to see how intuitive such an interface might be.

      • Use MRTK package for interactions

        • Using hand tracking creates challenges like menu access, no buttons, etc. Will try to find solutions. If not, controllers are a fallback option

    • To-Dos

      • Implement base functionality for a 3D modeling demo in VR

        • Changing object colors

        • Adding primitives

        • Toggle between edit and object modes

        • Moving vertices

      • Integrate MRTK for hand tracking in demo

      • Integrate Photon for collaboration

      • Assess strengths of such a project. Are there any reasons this might not be a desirable tool?

• Project 2 Milestones

  • 3/18 - Solidify plans for final project, start experimenting

  • 3/23 - Have hand tracking + manipulating object vertices + changing object colors working 

  • 3/25 - Adding primitives, toggle between edit and object modes, try to set up Photon. If Photon progress is not promising, consider other existing software to show the class

  • 3/30  - Continue getting Photon set up for class activity

  • 4/1 - Clean up base functionality/testing

  • 4/6 - Have wikis for MRTK, vertex+object manipulation in Unity set up. Add best practices for VR to wiki

  • 4/8 - Continue fleshing out wiki, getting Photon working

  • 4/13 - Have Photon working, test that full functionality is polished

  • 4/15+ - Finish wiki contributions, clean up project, prepare final presentation+ final materials

• In-Class Activity

  • Being able to host the project and have class join one server to collaborate together would be ideal. Not sure of specifics on how to do that in Unity yet

• Deliverables

  • Prototype demoing collaborative 3D modeling techniques with hand tracking enabled

  • Create/add to wiki pages on how to get these technologies up and running, best practices, and other details

  • Pictures, video, open source code that other students might want to build off of in the future

3/16/21 - 4 Hours

• Got hand tracking working

• Lots of experimenting on how to manipulate meshes at runtime in Unity

3/17/21 - 2 Hours

• Wrote up proposal in doc, linked in journal

• Made project 2 presentation

• Evaluated others' proposals

Project 2 Proposal Evaluated by Laila

• • The proposed project clearly identifies deliverable additions to our VR Software Wiki (5)

  • The proposed project involves collaboration in VR (5)

  • The proposed project involves large scientific data visualization along the lines of the "Data Types" wiki page and identifies the specific data type and software that it will use (4)

  • The proposed project has a realistic schedule with explicit and measurable milestones at least each week and mostly every class (4)

  • The proposed project includes an in-class activity (5)

  • The proposed project has resources available with sufficient documentation (5)

3/20/21 - 4 Hours

• Getting vertex editing working with hand tracking 

• Tests with color picker for mesh color changes

3/22/21 - 5 Hours

• Vertex editing working with hand tracking

• Color picker working

• Read paper on Seven Scenarios

• Started work on adding primitives

• TODOs (in order of importance): Add primitives, integrate Photon, toggle between object and edit modes (?)

3/24/21 - 2 Hours

• Works with controller input as well as hand tracking

• Recorded demo video

• Had roommates try demo 

  • Feedback:

    • Environment too dark/desolate

    • Might want tutorial on interactions

    • Hand tracking was a bit finicky (may be because of lighting in the real world space)

3/28/21 - 2 Hours

• Tested out Engage with George and walked through some of his visualization ideas for his in-class activity, gave feedback

3/29/21 - 5 Hours

• Tried to implement adding and editing new primitives, but there were lots of conflicts with the way I implemented vertex editing

  • Thinking there might not be enough time to implement this

  • Implemented toggling between Edit and Object Modes (or both at once) instead 

• Built scene to .apk file and walked through process to get it set up on Quest 2 using SideQuest

• Cleaned up scene based on feedback, bug fixes

• Started working on Multiplayer functionality with Photon

  • This doesn't look like it'll be ready for activity-- will have class download scene and use it in singleplayer, communicating over zoom similarly to the CavePainting activity

3/31/21 - 5 Hours

• Cleaned up scene, built APK, uploaded to Drive

• George helped me test the setup process-- got the program working on his Quest in ~5 minutes 

• Wrote up in-class activity with tutorials on setting up sidequest, getting dev mode working, setting up hand tracking in quest 2, and sideloading apk files to quest

• Wrote survey for activity 

4/1/21 - 3 Hours

• Looked at feedback in Google form from in-class activity

• Considering evaluations and key points for wiki

• Worked on Photon integration