VRPoetry

Yuanbo Li, 2023 Spring

VRPoetry is a VR application developed by Yuanbo Li under the advise of professor David Laidlaw. The full functional app can be downloaded here.

1. Introduction

For scholars of East Asian literature, the emotional elements present in Chinese poetry have long been a subject of fascination. One reason for this is that Chinese poets use diverse vocabulary and expressions to convey their feelings, and sometimes adopt pseudonyms when composing their verse. To effectively express emotions in their work, it becomes crucial to compare and contrast the various techniques used by different poets.

Virtual reality (VR) offers a unique opportunity to explore and analyze larger datasets that would be challenging to visualize otherwise. VR also provides users with a 3D system, which enhances the immersive experience. In this paper, we investigate how VR can enhance literature research by developing a VR application called VRPoetry, which groups Chinese poems in Oculus Quest2.

VRPoetry offers a novel framework for comparing poems based on the emotions they express. By categorizing input poems and assigning them numerical vectors that denote the intensity of their emotions, poems are arranged within the framework based on their emotional content and authorship. This framework aids researchers in analyzing poems, as it allows for easy identification of connections and variations in the ways emotions are expressed throughout the poems.

2. Related Work

Project Gutenberg is an extensive digital library comprising of more than 70,000 electronic books that can be accessed free of charge. These ebooks are available in various formats, including ePub and Kindle, and can either be downloaded or read directly online. By browsing through the available collection, it is possible to locate a dataset of Chinese poetry translations that has been uploaded onto the platform. However, it does not provide an emotional grouping of the works.

The Virtual Ancient Manuscript Exploration in Virtual Spaces project shares a similar objective to our own, in that it involves the adaptation of existing texts for use within a virtual reality environment. However, while our project is primarily geared towards facilitating the analysis of poetry, the Virtual Ancient Manuscript Exploration project is focused on recreating and preservation of ancient works, rather than their analysis.

The Virtual Rosetta project is also involved in creating a new visualization environment that utilizes virtual reality technology to explore historical drawings. Specifically, the project focuses on a collection of early twentieth century Vietnamese drawings, and employs a technique called hierarchical clustering with sentence embeddings to organize the drawings based on their textual descriptions. This technique involves mapping sentences to numerical vectors for the purpose of identifying similarities and differences among the drawings. In contrast to our own work, which focuses on analyzing textual content, the Virtual Rosetta project centers on the analysis of images and drawings. Additionally, while the Virtual Rosetta project uses hierarchical clustering to denote relationships among the drawings, we pair emotional strength with 3D coordinates in our own project.

3. Dataset and Classification

3.1) The Dataset

In our project, we have utilized Arthur Waley's translations of "The Poet Li Po" and "A Hundred and Seventy Chinese Poems - Chapter 2, Po Chu-I" as our dataset. We have chosen to use Waley's translations because Chinese poetry often does not express emotions directly, and requires careful reading of context and objects to decipher the underlying emotions.

For example, in the poem "The Flower Market" by Po Chu-I, the author spends most of the poem describing the beauty of the flowers, while briefly mentioning that "a cluster of flowers is worth the tax of ten poor houses." The deep-seated disgust for social inequality and the extravagance of the upper class is hidden beneath the surface of the original work, but Waley's translation successfully captures this underlying sentiment. If we were to analyze the poem using a Google translated version, we would find that the author appears to simply be "happy" and "pleased" with the beautiful flowers. However, by utilizing Waley's translation, we are able to discern the disgust and sadness conveyed by the poem, which is essential for our analysis of the emotional content of these poems.

3.2) The Emotion Classifications

In our project, we have classified emotions into six categories, namely anger, disgust, fear, happiness, sadness, and surprise, based on Paul Ekman and his colleagues' cross-cultural study from 1992. We then utilize the GPT-3.5 model to rank each translated poem against these six categories. Each poem is assigned a numerical value denoting its strength in each emotion, and we normalize the sum of these rankings to equal 1. This approach is grounded in our belief that emotions are multi-dimensional and complex, and cannot be reduced to a single dimension or classification. By utilizing multiple categories and ranking each poem against these categories, we can gain a more nuanced understanding of the emotional content of these poems.

While some poems express multiple emotions, we found that no two emotions are entirely contradictory. For example, Li Po's "Drinking Song" expresses both happiness and sadness. Despite the poem's focus on drinking with friends, Li Po wrote it during a difficult time in his life. Waley's translation effectively captures this emotion, subtly conveying Li Po's desire to "drive away the sorrows of a thousand years" through his drinking. Therefore, we have created corridors dedicated to each emotion, even if some poems express multiple emotions. This approach enables us to compare how a specific emotion is expressed across different poems, with poems that express multiple emotions appearing in multiple corresponding corridors.

4. Design

4.1) The Overal Strucrture

Our hypothesis is that a spatially-economical and legible reading environment for literature can be achieved by designing a structure that reflects the emotions classification and has walls displaying panels of poems. Our project's design features an altar at the center, which serves as a display area for information about the datasets. We have also created six separate corridors, each dedicated to one of the six emotions we have classified.

In order to determine the optimal layout and orientation of drawings and their accompanying labels, we experimented with several metrics. These included identifying the best grouping strategies to ensure that poems with similar emotions are placed close to one another. We also explored how readers can compare how different authors express similar emotions. In addition, we aimed to make it easy for viewers to compare the English translated version of the poems with their original Chinese characters that express emotions. Finally, we examined the influence of corridor size on the reader's understanding of the strength of emotions in the works. Through these metrics, we hope to provide an intuitive and comprehensive reading experience for users of our VRPoetry application.

4.2) The Altar

In our project, we have designed the altar located at the center of the structure to display key numerical data for the two datasets, including the total number of poems and the number of poems for each emotion category. The altar has been specifically designed to be elevated above the ground level, requiring users to ascend nine stairs in order to reach it. This design serves two purposes: firstly, it corresponds to the "magic number" 9 in Chinese culture, which is considered to be a symbol of completeness and perfection. Secondly, the elevation of the altar allows users to easily observe the data from a distance.

4.3) The Corridors

As previously discussed, we have created six separate corridors, each corresponding to one of the six emotion categories. We have also implemented several key features to enable a clear comparison and reading experience:

The walls: We have hung all of Li Po's poems on the left-hand wall, and Po Chu-I's poems on the right-hand wall. The walls have been carefully designed to be at a distance of 135 degrees from a viewer standing in front of them. This allows the viewer to focus on reading one poet's work while being able to easily turn to the other poet's work within two seconds.
The ranking: We have further divided the strength of emotions into three categories - weak (<0.4), normal (>0.4 and <0.7), and strong (>0.7) - and have arranged the poems in order of their emotional strength. Weaker emotions are placed closer to the entrance, while stronger emotions are placed further away. This allows readers to examine the progression of emotions within a single poet's work, as well as compare similar levels of emotions between the two poets.

4.4) The Windows

In our project, we have designed a 3D object for each poem, which is displayed alongside its Chinese original text on the left-hand side. The 3D window provides a museum-like viewing experience for the poem, while the yellow background creates a strong contrast with the black context, making it easy to view and read. This design allows users to engage with the poem in a more immersive way, and provides a visual representation of the emotional content of each poem.

6. Multiplayer

One of the advantages of VRPoetry is that it allows multiple researchers to work within the same program, making it a collaborative tool for literary analysis. Additionally, our coordinates correspond to emotions, allowing users to send coordinates to other users and teleport to a designated place. This feature enables viewers to view the same works together, facilitating collaboration and discussion among researchers.

By enabling multiple researchers to work together in the same program and allowing for easy navigation within the virtual space, VRPoetry promotes interdisciplinary research and collaboration, expanding the possibilities for literary analysis in the virtual reality setting. (Under construction)

7. User Studies

Background:

We have surveyed 15 students (11 undergrads + 4 grads) from Renmin University and Peking University in China. All of the participants are pursuing a degree in Chinese Literature, China History, or related. They are all fluent in both English and Chinese, and they have read many Chinese poems before the survey.

We offer each of the participants a 5mins brief tutorial on VRPoetry, and they each have 10mins using VRPoetry. After that, each of them have 5mins to answer the questions 1-4 (verbally + filling a form). Finally, each of them are given about 5 mins for work on question 5. Questions and results are originally in Chinese, this is a translated version. Surprisingly, a large proportion of participants are not in favor of using VRPoetry. Here are some reasons they provided:

Question 1:

Do you prefer doing research using VRPoetry over using screen monitors? If not, what’s the major problem?

Surprisingly, a large proportion of participants are not in favor of using VRPoetry. Here are some reasons they provided:

1)Participants can not take notes inside VRPoetry.

This means they need to take off the headset and wear it back very often

2)Participants find wearing a large headset for long time is painful.

3)Participants often need to search for extra information online during research. This is not enabled in VRPoetry. 4)When working on poems, especially the original texts, participants sometime want to copy and paste some part of the text, this is not allowed in VRPoetry

Question 2:

Will you use VRPoetry as a searching App to find a specific poem? Why will/ will not you use it?

Near half of the participants answered yes to the problem.

Reasons why use it:

1)The app is well structured, it divides the poems into sections.

2)Sometimes the researchers cannot remember the name of a poem, but they can remember its general idea. VRPoetry comes to be handy at this situation.

Reasons why not use it:

1)The dataset is too limited. I cannot find poem I want to

2)To find a single poem, participants need to walk through each of poems. That’s a waste of time

3)Walking and reading feels dizzy.

Question 3:

If not for doing research, will you read poems in VR?

Most researchers answered no to the question. They believe reading poetries in VR does not add anything to the experience. Furthermore, more than half of the participants believe reading poetries on paper/ books is the best option.

Question 4:

(Follow up to Q3)

If there’s something VRPoetry can do to change your experience (for reading poetries in leisure time), what would it be?

1)Design some immersive experience. Make people feel in the world the poetries are in.

2)Some technologies that can help me find poems quick. I want to quickly find what I want to read (this cannot be achieved by books)

Question 5:

The participants are given the short poem “Drinking together in the mountains” by Li Po, both English version and translated version. They are asked to read the poem both on screen and in VRPoetry. I record the second they spend to the read it.

The participants are slightly slower reading in VRPoetry comparing to reading on screen, but no big difference in general.

Some of them reported that they need to move their head when switching from the Chinese version to the English version in VRPoetry. One participant (who spend 34s in reading) report that she got lost when reading in VRPoetry). She reported: “The font size is small and the text are stacking when viewed from an angle”

Survey Conclusion:

There are certain ways to improve VR Reading experience based both on the answers of the participants and inference of the author.

Dataset based:

1)Include more poems (larger dataset)

2)Provide a more easy search method. (Readers do not need to walk and read all the poems to find what they want)

Reading Experience based:

1)Develop eye gaze tracking to help researchers understand where they are and avoid getting lost

2)Allow interact/ taking notes inside VRPoetry

Research Experience based:

1)Allow connection to the internet (?) to search for certain extra information

2)Allow copying selected text to a clipboard (for citing the text)

8. Disscusion

There are certain challenges that our method may face in certain cases. One of these challenges is the overlapping of emotions. It is common for poems to have the same emotional scores, but we cannot overlap these works in the 3D space. One solution to this problem is to make the coordinate-emotion link less accurate, which would involve placing two works with the same emotions close to each other in the same corridor, but not in the exact position.

Another potential challenge is the unbalance of the dataset, which could become a problem when the dataset is large enough. In the current example, Po Chu-I wrote a large number of poems in "sadness" (19) and "disgust" (13), while only a limited number of poems in "surprise" (3). If the dataset were even larger, then we could encounter a large number of empty spaces in the "surprise" corridor, while the "sadness" corridor may become overly crowded. Addressing these challenges may require further refinements to the methodology and data analysis techniques used in VRPoetry.

9. Conclusion

VRPoetry is a method that enables the comparison of two sets of data with certain characteristics within a virtual reality environment. While we have used this technique specifically for analyzing Chinese poetry based on emotional content, it can be applied to other types of data and grouping beyond emotions. In future works, we plan to utilize this technique in a more broad and diverse range of applications, allowing for the analysis and comparison of different types of data in a virtual reality setting.

Page updated

Google Sites

Report abuse