As discussed in the Viacom visit, our original goal of the visualization was to put the animation in three frames, and attach and sentiment data to the body, which will demonstrate the different engagement level between participant with different experience of VR, and the impact of different headset. However, as we went further, we found that it is quite difficult to integrating the physiological data and the movement in three.js in a proper way, so we decide to visualize a simple physical motion without attaching the brainwave and heart rate, and the visualization of motion trial dots. Also we will visualizing the sentiment and heart rate data separately from 3D animation in a more traditional way by using pie chart and real-time heart rate figure and sentiment heat map.
Our prototype, which has comparison across headset, experience level and stories, with attached heart rate
One simplified visualization we have now is the motion trail dots in the 3D space consist of the x,y,z value of three joints from the motion capturing system, it does not contain skeleton the the animation of the entire body, but it has the color represent the heart rate, and user can change the view by rotating the cube.
3D scattered motion trail plot, Github link is here
Another motion visualization done is by using the BVH file, in this visualization, viewer can choose the physical movement over different stories, it provide a complete animation, but the comparison is not presented at this stage.
Apart from using the physical motion data, the heart rate, and sentiment data over time are also a valuable source to visualize. Although heart rate is not the only indicator of the sentiment and the engage level, it is interesting to visualize it to see how different headset and experience level will impact the heart rate. Below is the graph showing the real-time heart rate trend between two participants wearing Cardboard and Samsung gear, one participant is familiar with VR while the other one only tried it few times. As can be seen from the graph, between the two participant with different experience level, the reaction in terms of heart rate can be quite different, especially the story they are watching is quite interactive and unconstrained. But it can be noticed that, for same person with different headset, the overall trend of heart rate is very similar, the cardboard one sometimes is ahead of the Samsung Gear in the graph, which means the participant generally respond quicker in cardboard, this might accord with founding in the AP report that, the cardboard elicited the highest level of stimulation, which is associated with individuals being more attentive than related.
The sentiment change with time for several participants were recorded in the csv file, so we decided to visualize a real-time figure to show that how the sentiment change during the story for three participants, who has different experience of VR. As the figure below shows, for the person that has little experience with VR, the sentiment changed way more often than the other participant, and the majority of the sentiments are red and orange, which means simulating and powerful, this is totally explainable because when trying equipment for the first time we are always excited and the sensitive. For the participant that is very familiar with VR, the “fascinating” sentiment is dominant and the change of sentiment are smaller than the other two It’s probably because when people are familiar with the tools, they will pay more attention to the story content, which lead to high attention level and fascination.
We’ve also use pie chart to compare the sentiment between different stories and different participants, in this case, the percentage of certain sentiment correspond to the frequency they appeared during the recording. But at this stage, we haven’t put the “participant B” yet, because we cannot find another person who participant in the recording for all three of“Mosul”, “Elephant” and the “New Orleans” data. Similarly, forHTC Vive, it was mainly used to test the “into the blue” story while the other two devices were never used for the story, so it is impossible to compare the sentiment between three headset for same story or person with same experience. This is the main limitation when we try to make a complete and interactive sentiment comparison which involves three headset and different participant.