What is Virtual Reality & 360 Video? And How to Make Use of it?
Concepts & Approaches
The purpose of this paper is to demonstrate some of the aspects and potentials of Virtual Reality. and 360 Video. Some of the of the ideas & concepts presented come from the mere imagination & knowledge of the author. Many of which are not practically proven and can potentially be hazardous. In addition, the following content may not be suitable for some audience. So attention is advised.
Author’s Note: Although some of the design concepts and approaches mentioned in this paper come purely from my imagination, and so far my research did not bring up any similar experimentation or products using the same ideas, I acknowledge the possibility of someone else who might have presented those concepts and approaches before I have. Should that be the case, I shall be more than willing to use the required attributions and citations as appropriate.
I remember the first time I watched the movie The Matrix. I believe I was eleven maybe twelve years old back then. It was a summer night. We used to sleep on the roof of the house as houses had a very unique design where I grew up which included a solid flat top and people used to sleep there during summer nights. I remember crawling underneath the sheets staring at the night sky for a while before I fall asleep. With so many thoughts & questions running through my head about the future of Virtual Reality. I was not afraid that a ‘Matrix’ will be created dragging everyone to live in. As a matter of fact I was feeling excited and optimistic, fantasizing a future where people can see more, experience more, and be able to travel beyond their bodies to anywhere on the planet and beyond. A world where people can be closer to help one another, live longer and spend more time for each other. Give more, and suffer less. Unfortunately back in the day I was just a child growing up in a what you may call a non-technologically savvy society. I turned my face to the right where my mom was sleeping. She decided to sleep in the living room where I stayed up late for The Matrix, asking me to wake her up when I’m ready for bed. She was worried to leave me all by myself around electronics because she knew that the times where I decide to conduct a reckless experiment were unpredictable. I don’t think she ever realized that I was more concerned and worried about her than she was about me. For I knew that the danger that was revolving around her was far more fatal than anything I could possibly get myself into. She was terminally ill, and there was absolutely nothing I could do about it except to pray for her and others like her every night, hoping to grow up and be able to do something to help them. Frankly medicine was, and still not one of my strongest points, but electronics and information technology were, and I hoped to be able to utilize that to improve sick people’s lives.
Almost two decades later, living in a country where ‘if there is a will, there is a way’, according to a remarkable college professor I’ve had the pleasure to meet a while back, I finally have the opportunity to help making lives better. I’ve spent the past five years or so filling up hunger for education & knowledge in preparation for the long journey. Not so long ago I was introduced to a motivational vlogger by the name of Lowi Sahi, who simply makes videos about his daily life living in the beautiful city of Dubai in the United Arab Emirates. As simple as his video blogs are, Lowi has managed to gather a sizable audience. Recently he has shared the motive behind his decision to share his private daily life with the rest of the world. There was a girl he said, a terminally ill girl who asked him to make a video about Dubai, a city she’s always wanted to see but could not due to her illness. Somehow, this has evoked strong flashbacks about where it all started for me: The little boy watching the stars, and of a better tomorrow.
I remember my first VR project. It was literally a cardboard box incorporating a sticky note book with flipbook animation drawn inside of it by yours truly. In order to see watch the animation, one would have to hold the box with one hand and use a slit on the top to play the animation manually. The biggest design flaw was that it was dark inside the box and you could not see the animation without illumination when the animation was being played. My solution to approach this problem was to incorporate a small light bulb extracted from a toy car I received for my birthday. Eventually I was able to extract the small, battery-operated motor of the car and employ it to play the flipbook animation by a push of a button. It was a gift I’ve made to entertain my mom when she used to lay in bed for days when she started a new chemotherapy session. I do not think she ever fully figured out how to use my ‘ingenious’ product because, well, it was not exactly rigid given the material used to make it. So let’s just say that the system required a lot of maintenance. I did not complain much about as it has given me more time to spend my with my mom.
Today, the world looks so much different from the one I used to live in back then. From supercomputers through smart phones, drones & social media and the internet. In my opinion, the world has never been closer, and the opportunity for innovation is much much bigger. But when it comes to Virtual Reality I believe we are still making baby steps, though I believe we are right on the verge of a whole new generation in the information era: The evolution of Virtual Reality, which would not be possible today if not for what, in my opinion, is one of the human’s greatest inventions, the digital camera. Here’s a little demonstration on how basic digital cameras work by James May, who I guess is doing informative videos now instead of driving Ferrari’s & Lamborghinis for a living…
But really, let us think for a quick minute, what can Virtual Reality really do for us? After all, it is merely video transmission. How far can we really push that? I have been asking myself this question during the past twenty something years. Starting my experimentation with a cardboard box with a small light bulb extracted from a toy and some flipbook animation projects.
What is Virtual Reality?
In order to understand the concept of Virtual Reality, we need to understand what video is to begin with. We all watch videos I presume. But how many of us have given it a thought of what video is really made of? Well, in case you have not, basic video is a bunch of still images brought together as ‘frames’ using light photons in order to give the impression of motion. It is just like flipbook animation but is done using a fast camera, also known as a video camera.
One of the limitations of this concept is that normal video only covers a limited Angle of View (AOV). Which is why when watching a normal video you can only watch it from a certain angle, basically a rectangular shape of what’s in front of the camera. The simplest way to put the concept of Virtual Reality, in my opinion, is by describing it as the incorporation of a 360 feed which again is a bunch of images sewn together, except this time it is forming a ‘sphere’ instead of a rectangle, covering a 360 angle.
The camera units used in performing the task would serve as the axis of the sphere. And so will you when you are watching the 360 video. Now that we’ve hit two birds with one stone explaining the concept of Virtual Reality and 360 Video, we can dive together a little bit deeper beneath the surface of Virtual Reality armed with those useful skills we’ve equipped.
So imagine yourself watching a 360 video using special goggles giving your eyes a direct live view from a 360 camera which is placed, let’s say Japan. Unless you’re in Japan, then you imagine some place elsewhere. But let’s assume we’re in Japan in Spring with cherry blossoms everywhere. Of course our physical bodies aren’t really in Japan but one of our senses, which is our vision, is seeing Japan in real time. And since this the video feed you are experiencing covers a 360 degree then you can virtually turn your head in any direction and you’ll be able to see that particular angle just like in real life. But what if you decided to take a step forward inside the virtual space? Of course nothing will happen unless the 360 camera that is broadcasting the live video moves with you. So what solution can you propose to this? Hire someone handle your 360 camera for you for wherever you want it to take you? You might as well use the fortune you are going to pay that person that if you were lucky enough to find one to move wherever you want yourself without the use of Virtual Reality altogether! But what if we can do something else?
Okay, do you remember we’ve talked about the 360 video concept as being a ‘sphere’ of images sewed together with one another? What if we could make the size of this sphere even larger by adding more images to it? And instead of having a static axis, why can’t we incorporate a mobile one? Again you are going to ask me how. Here’s how:
Let’s go back to the real world for a minute. Step outside and look around you for a few moments. Now, if you were to describe the shape of your surrounding atmosphere, or the universe in general, how would you describe its shape? Let me make it easier for you. Do you see any angles in the sky by any chance? I sure don’t! This means the shape of our world is nothing but, yes spheric! As you can see, we live in a giant sphere. Or at least that’s what is appears in the eye of the beholder.
But if what we’re experiencing in our real world is nothing but images, how can we see things in a three dimensional way while images should only appear in 2D? To make this concept more clear I need you to look at an object sitting next to you. It probably does look like it has three dimensions, but if you don’t move your head and keep it still in any given angle or ‘frame’, what do you see? Is it really 3D? Or is it only a two dimensional image? You see, our eyes were built to adapt with the current of view. It basically gathers information about the AOVs of objects forming an impression of a three dimensional map to help us get around. As simple or complicated as you can possibly imagine.
Now, after this nice virtual picnic we just shared outside, let’s get back inside our Virtual Reality goggles. How can we expand the size of the little sphere we’ve created using our 360 camera? (Of course without having to hire someone to carry it around for us all the time.) The answer is to add more cameras. But how will this work out? Let’s have a thought.
P.S. You may want to read the following multiple times because things are about to get a little complicated.
Imagine a sphere built inside the virtual world using a software similar to the ones being used to create video games with the ability to function inside of a server. I shall call it ‘The Virtual Sphere.’ Now, let us set up a small studio in a room somewhere. We’ll bring several 360 cameras and position them to cover every single angle of the room. When we press the record buttons, the video gathered from all of the cameras will be transmitted in real-time to a special software which I shall call ‘3D60 Maker.’ What the software does is to grab the images coming from the live feed in what I shall call a 3D60 video and paste them on the inside layer of the Virtual Sphere file.
So, what do we accomplish by doing this? The answer is, not only we will be eliminating the problem of moving around inside of the sphere because we can now simply create a virtual axes similar to those in video games (which we can even use to usual things such as flying in Japan instead of simply visiting it in real-time because, well, it’s the Virtual World!) But we can also share that experience with as many people of as we like because now the 360 cameras will be uploading the data to a software inside of a server instead of transmitting it directly to the client. And of course, we can now mount the cameras anywhere we are allowed to not just in that studio room, which was only an example. Problem solved!
By now, we should be able to virtually travel to any destination in the world using our Virtual Reality goggles, given of course that there is 3D60 coverage in that location. And now only we can travel freely within that 3D coverage, but we ourselves can travel in 3D mode. Meaning we can fly and rotate however we like as if we were in a video game, which in a way we will be. But is that all we can do with this technology? Oh my friend… I was just warming up here!
Let’s take the concept we have from earlier and add a little bit of spice to it. The Virtual Reality experience we have just discussed is using video feed, and as you’re probably aware, video can be recorded, and by recording it, you are simply taking a piece from the past with you.
Have you ever had a deep thought of the concept of video recording? I think it’s fascinating and creepy in a way. No wonder why when early video came out some people thought it was made by the devil. It’s like a time machine which you can use to travel to the past, except you’re only allowed to look through the window without being able to change anything. Not that we want to because only God knows what can change in the present if we altered the past. Bad idea. Except I don’t think time works in the way people imagine. But there’s no worries here. Because we’ll only be altering with the small ‘copy’ of a video which we will have inside of a closed independent reality system. More like another dimension or an ‘alternate universe’. So it will be as risky to our existence as editing a normal video.
So let’s take a 3D60 clip that was recorded say 24 hours ago. In the video we see grass, a sunny sky, and a person who’s expressing a weird smile. Of course we should be using our goggles in order to experience the scene from a first person view. But you know what? I passionately hate the ugly bulky shape of Virtual Reality glasses, so why don’t we replace them with something more practical and stylish? Say contact lenses. After all the only limitation we have in this demonstration is our own imaginations.
Yup Yup! Now we’ll be wearing virtual lens to experience the Virtual World. How awesome is that?! Perhaps that’s why the person we see inside the virtual world is making that weird smile. Or maybe he’s just weird like that. But anyways, since we’re in control, we can now alter that scenery in any way we want! And keep in mind that this video has been recorded 24 hours ago so everything in that scenery could be different now. Including creepy person. Are you thinking what I’m thinking?! Let’s pour some rain on him!
Now that it’s raining on creepy person, should we expect that his facial expression might change? Of course not! It’s just a video. And that person is probably somewhere else now. Unless he’s still there making the same weird smile. That would be seriously creepy… Anyways, let’s not get too far from our subject. So basically when we can take a feed from the past and edit it however we like. Except the effect will only occur in the target object. But what if we can add some more spice to it and basically create ‘rules’ for everything inside of that video to edit itself based on ‘cause and effect’ principles?
There are two ways to approach this that I can think of. One would be to design our own rules by writing them as a code, again like in video games. In other words we can program the creepy person in the model to change his facial expression when it rains. Or, we can take it to the next level make use of our new friend AI to make things a whole lot more exciting, and possibly a bit creepy…
AI, also known as Artificial Intelligence, is a term used to describe rational behavior learned and used by machines in order to achieve a certain goal. You can learn more here: http://hooterbay.com/2016/06/13/artificial-intelligence/. In other words it’s like what we call ‘thinking.’ For our cause, we’ll be giving our machines eyes and teach them how to study and analyze what they are watching, pretty much like we humans do. Except whether those machines can develop a form of consciousness like ours or not,I cannot tell you right now and it’s a totally different subject. But let’s stick to the idea, imagine constantly feeding 3D60 video to a computer with software installed that teaches the computer how to analyze and gather data about the behavioral traits of the target object, or person.
Allow me to elaborate a bit further, let us assume that we live in a world where everyone is wearing VR lenses which record everything they see, and there are also incorporated 3D60 cameras constantly mapping the exterior space, or whatever space the user allows it to. The recording lenses will be building a memory profile for that person, and the 3D60 cameras will observe how the person will react towards what they are experiencing. The data will be sent to a central computer system (CCS) with AI capabilities that will analyze and build a profile for this person. Not only the memories of the person will be stored, but also the personality and behavioral traits will be captured. Let’s also throw in some other types of sensors to gather additional data. I shall call this array of data Electronic DNA, or ‘EDNA’. Whether it was the sky, the grass, or even creepy person, or anything else. After a while, the artificial intelligence will be able to mimic, and possibly even detect the future behavior of the target object to an extent. Of course according to quantum physics it is virtually impossible to accurately and completely predict the behavior of matter every time. Although, it might be possible to initially predict some behavior. Think of it like weather forecast or psychology. When you learn about something, you may be able to predict a significant percentage of its future behavior. Now, I am not saying we should use this technology to clone people inside the Virtual Universe, but just think of all the potential of being able to not only play any scene from the past, but also have a record which may be useful to extend, or renew the human lifespan at some point in the future.
However, in our current case scenario here we will simply be playing with video feed. Nothing more or less. So if the 3D60 technology with the use of Artificial Intelligence was able was able to generate a EDNA profile for creepy person, would it be possible to predict his behavior towards the rain and apply it as a new frame to our Virtual Reality experience? I think it’s just a matter of time before we see this concept functioning.
So far, we’ve learned about some things we might be able to accomplish in Virtual Reality. But none of this can compare to the experience presented in the movie The Matrix, and honestly to me Virtual Reality is not that excited without being able to at least touch the object you’re seeing, but would that be possible given what we know by now about VR? Let’s find out!
I can always go about futuristic technologies such as manipulating the brain in order to give it an impression of senses, and even though I’m thinking that would eventually be possible, I think it needs substantial amount of time, which is why I am going to skip it and discuss some concepts which we may be able to achieve sometime within our current lifespan.
Let’s talk about touching. What really happens when you touch something? Well, according to this video, as crazy as this may sound to you, we’re not really touching anything! But rather what happens when you go to touch something is that the electric field surrounding the electrons within your body repels that of the object you’re touching. Crazy huh? Let’s watch the video.
For us VR enthusiasts, this can be great news. Because now we know the ‘real’ experience of touching is nothing but electricity. And electrical behavior can very well be replicated. So let’s create another model really quick. Imagine a thin layer of wireless sensors attached to your fingertips, and those sensors create a virtual model of your hands inside the Virtual Universe. The sensors will also be programmed to give a small electrical impulse which happens when you interact with an object inside the Virtual Universe. Not only that, it can also alert the object you are touching, whether that object corresponds to a real life model or not, allowing you to interact with it in a similar manner to that in the real world. There you go, this was touching for you.
Bear in mind that in addition to the general concept of Virtual Reality there is also a concept called ‘Augmented Reality’ which is basically adding elements from the Virtual Space to the real world. Imagine how cool would that be if it incorporated the technologies described above? There are also potential ways I have in mind to replicate other senses, and potentially even replace dreams with a Virtual Reality, but I believe that the main idea has already been conveyed.
Here we come to the last and most important question of all: Do we really need Virtual Reality and 360 video? I mean, we’re already living in the real world. How logical would it be for us to try to replicate our world? Or add elements to it which although may give a similar impression but they’ll probably never feel as ‘real’ as the real world. My answer to this question is this:
Imagine a world where people can be anywhere anytime? A world where ‘The Matrix’ is not some secretive conspiracy that not many people know of but rather is something everyone knows and makes good use of? Imagine a world where physicians & surgeons are able to provide a prompt response to those in need wherever they were? Imagine a world where we can even be productive in our sleep, the whole third of our lives we spend hibernated? Imagine a world, where a girl like the one who inspired Lowi Sahi to start his vlog series could travel anywhere she desires and receives the medical attention she needs? And finally, imagine a world where we can all live together as one mind. With potential to conquer anything, even death itself.