Realistic Basic Realistic Content Planning
GCC Academy
Learning Goals Can explain the development process of virtual reality and augmented reality through movies Can explain the background of the emergence of frame-expanded format through VR/AR history
Can explain the planning process through examples of content Learning contents Movies that implemented virtual reality and augmented reality History of virtual reality and augmented reality Planning process through content examples
CEO Jeong Jiwon Career Current director of T.OZ Former director of WYSIWYG STUDIOS General Manager of TaekwonV VR for Lotte World General Manager of GyeonggiVR AR for Gyeonggi Content Agency
General Manager of Realistic Video for Jeju National Museum General Manager of Realistic Content Production for Sokcho City Museum
I am Jeong Jiwon, here for the lecture of Realistic Contents
The contents that we are going to talk about today are virtual and augmented reality of realistic contents
When we talk about realistic contents,
the most mentioned areas are virtual reality and augmented reality
So we are going to explore the movies that implemented virtual reality and augmented reality
Then we will explore how VR and AR has developed over time
After that, there are contents that have been produced through virtual reality content
We are going to take a look at the production process through this example
Movies that implemented virtual reality and augmented reality
You probably watch a lot of movies
Especially when watching SF movies, there are various technical aspects implemented
For example, there's Ready Player One, a movie produced in 2018
I mentioned this before in my previous lecture
This movie has been produced by Steven Spielberg
The setting is in the year 2045
Year 2045 is now approaching pretty fast
2045 is depicted as very gloomy and depressing
Many people use HMD
and travel to an oasis of virtual reality
In this oasis, everyone has their own character or avatar
Using this avatar, they can go anywhere and do anything they want
Things that are not possible in real life,
the imaginations are made possible in this virtual space
There's the main character here, and as for this main character,
their only comfort is, well they're a well-known figure in this oasis
This person does many various things in this virtual reality
When the maker of this game died, they left a last will that
someone who completes a mission will receive huge inheritance
Following this will, many people participating in this oasis try to solve the mission
When you look at this movie, they put on an HMD in order to access the oasis
After putting on the HMD, in order to really feel like they're walking around in virtual reality,
there's treadmill for VR contents, which is a rubber board that moves
and when I walk on it, it really feels like I'm walking in the virtual space
As for this movie, the current techniques weren't overused
and were naturally implemented into the movie, and this is a big positive
Especially, in this movie, the main character puts on the HMD and a haptic device
The character can experience things with this haptic device that allows the body to feel a sense of touch
This haptic device can also be created through techniques that exist today
But in the movie, they're expressed in a much more detailed and intricate way
So their devices are more dimensional and of a higher quality than the haptic devices we use today
The important part when watching this movie is that
in order to enter the virtual place, in terms of technical aspects,
the wireless communication network is very important
We have 5G and 6G right now
But to be honest, in order to experience all these attractions in real-time,
the wireless network is very important
As you observe this movie, the main character puts on the HMD and wears the haptic device
It shows how they walk around the virtual world while wearing this haptic device
And the interesting scene is that
there's a scene where the male main character and the female main character dance
There, the female character touches the male character
and he feels it through the haptic device
So this movie is very meaningful because
the currently existed techniques aren't overused and implemented realistically
And the areas that are more advanced than reality are
the wireless communication network
So through this wireless network, they wear the HMD and connect instantly
Going into the virtual space, performing economic activities in there,
as well as having social life are meaningful aspects of this movie
The main character actually wanted the haptic suit very much
So in the oasis, the main character collects coins as currency,
and using these coins, they purchase the haptic suit in the virtual space
Then they can actually receive the haptic suit in real life
Things like these remove the sense of differences between reality and virtual reality
and make it possible to feel like they're experiencing one world
The next movie we are going to take a look at is Minority Report, a movie from 2002
This movie has also been produced by Steven Spielberg
The setting of this movie is 2054
The plot is about predicting a murder and arresting the soon-to-be murderers
through this newest safety system called Free Crime, in this SF action movie
This movie is meaningful because
when they were producing this movie, the things that will happen in the future of 2054
were very well-depicted
So when you see electric cars going about today,
the electric cars don't make sound
These things were depicted in the movie beforehand
So when you look at this movie,
there's a scene where Tom Cruise puts on gloves and arrests the murderers mid-air like this
It's a very famous scene
This scene presents holographic screen
Also, when Tom Cruise is being chased and he gets off the subway,
the scene shows that iris recognition is used to personally match advertisements to each person
These are techniques that are on the buzz today that are well-implemented into the movie
As I mentioned earlier, futurists participated in the production of this movie
And the technical aspects are such naturally implemented in each and every scene of the movie
So when you watch Minority Report once again,
there's transparent display and interactions based on motion detection
And the most important part of this movie is that
when the main character thinks about their dead son, the hologram projector appears
For this hologram projector, even in 2054, the hologram itself
isn't very vibrant and is depicted as a rough screen
But just the fact that it can be seen in the space
made it seem very realistic, as presented in this scene
Another thing to think about while watching this movie is that
in Japan, they made this hologram product called gatebox
There's a clear, cylindrical case
In this case, the holographical character appears
This character is the famous virtual idol in Japan
This is a product that allows them to interact with the idol
As for this product, they have a conversation through AI
and there's also a sensor, so when communicating,
there's no awkwardness at all
You've probably seen the movie
is also about a man who dates a virtual AI program
Scenes that were only possible in movies are now seen in real life today
which is a very interesting aspect
The next movie to look at, and when I talk about virtual contents,
I use this movie as an example very often
This movie was made by a French Director Luc Besson, based on an original cartoon
When you watch this movie, you may feel overwhelmed by the absurd setting and the excessive computer graphics
But the important part of this movie is that
there's a scene where the male and female character go off to find the crime scene
They ride the bus and go to a desert
There's nothing in this desert
You know how theme parks have an entrance? There's just a gate like that
There's just a gate, and they put on the HMD on the people
The HMD here is an HMD that allows them to see augmented reality
For earlier, it was VR
They used HMD that can experience virtual reality
The HMD in is for experiencing augmented reality
It's good to understand it as that
In this movie, there's a market called Big Market
It's a huge thing, and in the movie,
it's the size of 5-6 soccer fields combined together
When they use the augmented reality HMD, the foreign markets appear in this desert
So they depict scenes where they actually buy things
I think this part is the best example that shows augmented reality in movies
When we talk about hologram, we often talk about Star Wars
But when we talk about augmented reality, it's the Big Market in
It's the Big Market in the desert, but there's nothing
But once you put on the HMD, places appear
And it's possible to actually purchase things
When you watch this movie, you can probably
imagine the things that can be made possible in the future
[History of virtual reality and augmented reality]
So far, we talked about examples of how VR and AR were applied in movies
Now we are going to explore the history of virtual reality
The person who first used the words virtual reality
is Antonin Artaud, a French playwright/actor
This person described theater in his "Theater of Cruelty" in 1938
When describing the theater, he used the words "virtual reality"
But this term itself is a little different from
the digital-based virtual reality that we talk about today
Probably, since they used it in stages of England,
it's like a virtual world that contains artificial environment and a mix of reality
And that's probably why he described it as "virtual reality"
Also, what made the concept of virtual reality well-known to the public was
a company called VPL Research
In 1985, they brought out HMD and a glove-like device
They showed the concept of virtual reality for the first time
Next, you'll probably know when I talk about View-Master
View-Master was made in 1939, and what this is, is that
It's cylindrical at the side and there are films
So when we move the lever on the side,
we can flip through each image
So back when we couldn't travel to other countries,
it allowed us to see through these film cameras
For this as well, it can also be described
as the very first virtual reality
In 1957, Morton Heilig carries out the first patent application of HMD devices
For this HMD device, it's very similar to the HMD device we use today
So when you take a look at this application image,
You can't really tell the difference from the ones we use today
You can find high levels of similarities
Later, in 1961, the developer of Philco makes a system called telepresence
The telepresence uses the HMD method
to view videos in long distance
Through the outer video camera that moves according to the head movement,
they were able to watch the movie
In 1962, Morton Heilig also creates a personal 3D stereoscopic theater
The theater is called Sensorama
This Sensorama is important for stereoscopy
and a very important keyword when talking about virtual reality
During this time, other than watching videos,
it was also used for soldier training and teaching materials for teachers
This Sensorama was called the "Cinema of the Future"
The interesting fact is that this Sensorama is a viewing system for 1 person,
where they can view stereoscopically but they can also experience motion as well
There's motion such as the chair moving and shaking
There's also scent, wind, and stereo sound
It's the very first system that provided 4D and 5D that we talk about today
So when you go around and talk about virtual reality and 3D solids
if you walk about Sensorama, you'll probably feel very smart
In 1968, there was a scientist in the U.S. named Ivan Sutherland
This scientist developed 3D display
The alias for 3D display is very well-known
3D display made by Ivan Sutherland, also called Sword of Damocles, is in the form of HMD
But it's too heavy to be put on the head
Instead, they set up the device on the ceiling and used a separate display device connected to it
and they can watch it while standing
They called this HMD the Sword of Damocles because
the King of Sicily, Dionysius, saw that Damocles was envious of authority and wealth
In order to show him the danger and anxiety of having authority, he invites Damocles to the banquet
and seats him on his seat
Above this seat, he hung a sword up high, and that's how the name came to be
A characteristic of this HMD is that they used translucent display
And on the user's eyes, they put on the AR glasses covered in cubed graphic
They also used head tracking through the headset
All the systems of virtual reality display were all involved
and optimized in this device
In 1979, Eric Howlett
founded the company called Large Expanse Extra Perspective
He filmed a stereoscopic picture with chromatic aberration
and allowed this film to be viewed through its exclusive viewer
And in the process of the lens altering this chromatic aberration,
the system provided a very wide viewing angle
In 1984, there's a lab in NASA called VIEW
They developed an HMD here
In 1986, they also invented an HMD that
performs virtual visual environment display
In the 1990s, the lens-based HMD made by LEEP, the globe made by VPL,
and the remote camera made by Fakespace Lab all worked together
to create a new form of HMD
In July 1989, a virtual reality system called RB 2 was also released
At this time, the weight of this HMD was 2.4kg and the resolution was 320x240
And the price was 9,400 dollars
When we talk about virtual reality hardware today, we can't forget the Oculus-Rift
The person who made Oculus-Rift was
the CEO Palmer Luckey and the game developer John Carmack
These two used the gyro sensor of smartphones
and allowed them to view the video by turning their heads
So they can view the movie in 360 degrees
Also, in order to distort the screen caused by lenses magnifying the video,
they additionally produced rendering that distorts this distortion again
In 2014, as Facebook takes over Oculus,
VR content and techniques
started to fully reveal itself to the world
The next thing to explore is the history of augmented reality
There's a book that Frank Baum published in 1901
The title is THE MASTER KEY, and in this book, there are these glasses called Character Marker
When they wear these glasses and meet people,
characters that describe their personalities show on their foreheads
So if they're a good person, the character G appears, and for evil people, E appears
Wise people get W, foolish people get F
For kind people, K appears
It's very innovative that this concept appeared in a book published in 1901
When we talk about augmented reality depiction through glasses, we can't leave out Google Glass
Google Glass, an example of recognizing a person's face and showing information about them,
is an example very similar to the character marker
described by Frank Baum
When we talk about augmented reality, we can't leave out Boeing
As for Boeing, in 1990,
they carry out many research activities to improve the plane production process
The result was the wearable AR device, a concept of AR device
Plane production especially requires high cost
When they use actual, physical parts, they each require money
When it fails, the cost increases even more
In order to cut down on these expenses, they use virtual forms
and think about things they can do by using augmented reality
So back then, for plane production,
they made a wiring diagram according to each section on a giant plywood
They produced a particular plywood that contains information based on each inside area of the plane
The cost was very expensive
But when they used HMD glasses,
the wiring diagram of the inner plane shows on the plywood
So plywood re-use and grasping the location of plane work became very easy
If design changes were needed, the computer would edit the wiring diagram
and show then on the diagram
In 1992, Virtual Fixture was developed in the U.S. air force
For this, 3D graphics were placed to increase efficiency in their work
When the user moves their hand to control, the accuracy increases
Also, the guiding mode for movement was developed
which allowed precise work
So things like these, like the AR produced by Boeing,
as well as similar examples continuing to develop,
the techniques were upgraded
Especially, for this technique,
they not only mark the range and information of the surgery area,
but also how the surgical tools move
And in 1994, Professor Paul Milgram
defined the concept of Mixed Reality, or MR
This itself combined everything in the virtual and real world
into one term, mixed reality
In 1997, Ronald Azuma presented A Survey of Augmented Reality
Here, they provided a clear definition of AR and device structure
First, AR has to be a combination of reality and virtuality
Second, it has to interact in real-time
Third, it has to exist in a 3-dimensional space
All the things related to augmented reality that we're experiencing now
perfectly match with these definitions
The main buzz topic of augmented reality was probably Pokemon Go of 2016
As the game Pokemon Go was released, the concept of augmented reality also emerged
Many people would carry their smartphones to many areas
to collect items and gather points
Many other games were released after that
But when you look at Pokemon Go,
the virtual object of Pokemon were very well-mixed into the real world
As many items are involved in the physical space,
if you imagine a system where you can interact in order to obtain points,
you can probably understand that this concept is very well-implemented
When we talk about augmented reality, Colombia University in 1997
released an AR device and experiencing content called Touring Machine
Back then, the device was very heavy
Recently, backpack computers were made
but backpack computers back then were almost like actually carrying a computer on their backs
They had to carry very heavy devices on their backs
And many cables were connected additionally
In the beginning, when presenting augmented reality outside,
things like these negatively affected the cost and efficiency
But afterwards, based on these techniques, many more techniques were developed
Later, smartphones and glasses were used
for the production of many augmented reality devices
Recently, augmented reality were not only implemented in glasses
but in car displays as well
So you can see that they were closely involved in our daily lives
[Planning processes through content examples]
The next thing I'm going to mention is that, my previous company was VFX company
I produced VR contents there, which was Robot TaekwonV VR
Some of you may know, some of you may not, but Robot TaekwonV was
Korea's representative mechanic animation made in 1976
It's the very first domestic IP that used the concept of robots
As you can tell by the name Robot TaekwonV, they do Taekwondo in this animation
They fight off the enemy through Taekwondo
A major weapon is the robot's arm shooting out
Most of the time, the character does Taekwondo to fight the enemy
So using this Robot TaekwonV as the material,
I will talk about how I produced the VR content
Actually, although Robot TaekwonV was made in 1976,
many animations of it were produced afterwards
Especially in 2009, they carried out a real-life version of it
So top-class domestic VFX companies for movies and dramas gathered
to carry out this real-life project
Back then, you know the movie Transformer, right?
The concept designer of Transformer also participated
to change and modernize the design of Robot TaekwonV itself
If you look at it in a certain way, it was very similar to Transformer
Back then when we were planning Robot TaekwonV, we set the format to interactive walking VR
We set the experience time to 6 minutes
The hardware composition back then was HCC VIVE
We used HMD connected to computer
And just like the treadmill used in ,
we used walking simulators like those treadmills
and allowed them to wear the HMD and walk on the walking simulator
Then, we put on haptic devices on the wrists
and produced it to feel vibration when they hit with their fists
We set the experience time to 6 minutes because
VR contents back then were usually 5 minutes or less than 10 minutes
So the most common VR attraction was riding a roller coaster
as well as producing horror contents
VR was used very often
We set the focus on being able to experience the Robot TaekwonV VR within 6 minutes
And as for this content,
it's meaningful that it was the very first character match/fight VR that used authentic content IP
When producing this Robot TaekwonV VR attraction,
setting the concept of the character was very important
When producing VR contents or realistic contents,
people of all areas communicate in an organized way
It was very important that there were upgrades in every step
So the character concept was very important
When designing characters and converting them to 3D,
it doesn't just end by handing over the designed character to the 3D team
Since the characters are robots and there are enemy characters as well,
it's very important to think about how their movements can be more natural
So the character designer should not only focus on producing an exceptional 2D character,
but also think about the movements
and how they're going to carry it out mechanically, is very important
If they communicate with the 3D worker beforehand and carry it out,
a dense content can be produced
Especially for VR, the abilities of the developers are very important
For this Robot TaekwonV VR, unreal engine was used
Unreal engine and unity engine are most commonly used in VR content production
Recently, techniques of unreal and unity engine were highly upgraded
And they're used not only in games, but also widely in movies and dramas
When using unreal back then, when enhancing the video quality of VFX,
we felt that unreal was best optimized, so we made well-use of it
Back then, unity was more specialized for games
Unreal was specialized for video work such as VFX
We used unreal back then,
but our recent work uses a lot of unity as well
So when you produce realistic contents,
there are various 3D and development tools,
but these days, they usually mixed and used together
So for those in the 3D area especially, knowing about the game engines
will be very helpful
Especially when you're planning it as a career,
it will be very helpful for your portfolio work
As for Robot TaekwonV, you put on the HMD and use the left and right controllers
It allows you to punch with the fist
Next, since Robot TaekwonV walks around the city,
there's walking simulator like the treadmill
You can hold onto the sides and walk
But since you need to use the controllers,
the guides on either sides
are there to make sure that you don't fall over to the sides
We also made the story line of Robot TaekwonV experience
Before the actual experience begins,
there's content called cinematic
Through cinematics, they can take a quick overview of how what the content is composed of
Then after learning how to play the game using the controller,
they go on to play the actual interactive game
To tell you about the overall production process once again,
the planning team creates the scenario
They produce the scenario, then
using this scenario, the concept or design team
designs the character, the background, and various object designs
Using these, they create the storyboard simultaneously
Along with the storyboard, they produce the continuity
Using this continuity storyboard,
the 3D team produces the modeling, mapping, and animation
At the same time, the development team carries it on to HMD
so that they can actually be experienced
To organize the steps of Robot TaekwonV attraction once again,
The cinematics come out first
The cinematics introduce TaekwonV first
After the cinematic film ends, they can experience the first engagement game
After the first engagement game ends, a cinematic film plays that introduces the middle boss
After this cinematic film, the second engagement game can be experienced
Lastly, after watching the cinematic film that introduces the final boss,
the content is composed of battling the final boss as the last step
So in the first stage, TaekwonV first appears from a hangar
as one of the presented processes
And in stage 1, it comes up from the hangar and dispatches to ground floor
When it dispatches on ground floor, various
enemy characters fly towards it, and when they do,
I can lift the controller and fight them off with my fists
The good part when experiencing this, was that
I'm actually walking and performing action with my fists,
many experiencers felt catharsis
through these parts of the experience
In stages 2 and 3, there are moments of fighting with the enemies
They fight with the middle boss and the final boss
Back when we were producing this, the most important part was that
when we use HMD, we need to experience it while looking around 360 degrees
Therefore, the UI telling the user where to look was very important
So especially when the enemies were appearing,
well, when aircrafts fly towards them,
they can just hit them with their fists and get rid of them,
but when the mid- and final enemy characters appear, the move sideways
So when they don't know where to look,
the sound system is used to tell them the location
to lead them naturally into the correct experience
This is important because when we present on a rectangular frame,
there's a certain area that we have to present inside of
But as for VR, the full 360 degree view is shown
so it's very important to present where to look
So if you look at recent VR contents,
you can tell that they put a lot of attention on the UI
Before, for VR contents,
they didn't really consider people's viewing angle while producing the content
But recent VR contents put a lot of consideration into UI and UX
As soon as they put on the device, they can tell
where to look and what to experience
So when we made this content, in terms of UI and UX,
the text, images, and sounds were carefully placed
so that the experiencer can adjust their view accordingly
The unfortunate part about producing this VR content was that
as I have mentioned, it involves Taekwondo
Taekwondo basically requires fist punching as well as kicking,
but back then, kicking was not possible
Many people back then also said that they wanted kicking to be involved as well
It was a bummer that it couldn't be utilized
But basically, through fighting actions using the fist,
this content was pretty effective in inducing a feeling of catharsis
Also, the interaction part was probably the gaming aspect
Because if I just simply crush that,
I can't really understand the full situation
So in general, we set the target
and give points if they hit the bullseye
And when the content experience finally ends,
it has been set so that users can tell their final score
When we first created the VR attraction,
there was no point system
But as users experienced it,
they suggested that point systems would be nice, so that's why we added it
And when we added it, the reactions were much better
We explored a lot about VR and AR contents today
As for the VR contents,
I told you about the keyword, sensorama
For this sensorama, it not only implemented stereoscopy,
but also sound, vibration, and smell
Then, as the HMD called Oculus-Rift was released, the publicization of VR began
Afterwards, the camera-based 3D space allowed them to
track their own location and the users' locations
And nowadays, even without connecting to the computer,
the HMD itself contains all the needed sensors
So it recognizes the space, and within the space, the interaction of the objects
were carried out in this way
One thing I wanted to emphasize once again about VR contents is that
when you produce VR contents,
the location of the audience becomes the location of the camera
Therefore, the camera has to move in the audience's perspective
So unlike the existing video presentations,
the direction of the camera has to be decided, and how the character appears
Things like these have new presentation methods
Think about these carefully
and use them well when producing VR contents in the future
Summary Movies that implemented virtual reality and augmented reality Ready Player One (2018) Oasis: virtual space that makes impossible things possible
Makes it possible to experience real and virtual world as one space
Minority Report (2002) SF action movie that involves a free crime that arrests soon-to-be murderers Depicts the future of 2054
Valerian (2017) Movie for imagining what will be made possible in the future
History of virtual reality and augmented reality History of virtual reality Founder of the word, Virtual Reality: Antonin Artaud, the French playwright/actor 1939: Viewmaster emerges 1957: HMD structure patent application
1961: Telepresence development 1962: Personal 3D theater, sensorama development 1968: 3D display development 1979: LEEP company founded
1984: NASA's VIEW lab develops HMD 1989: Virtual reality system RB 2 released
1990: LEEP, VPL, and Fakespace Lab works together to create a new form of HMD 2012: Oculus-Rift production
History of virtual reality and augmented reality History of augmented reality 1901: Character Marker appears in the book The Master Key
1990: Boeing develops wearable AR device for plane production process improvement 1992: U.S. Air Force develops Virtual Fixture 1994: MR is defined
1997: Definition of AR and device structure is presented 2016: The concept of augmented reality is known through the Pokemon Go game
Planning process through content examples Robot TaekwonV VR Format: Interactive Walking VR Experience time: within 6 minutes
Composition: VIVE Walking Simulator Controller Unreal engine 4 Character match/fight VR using authentic content IP
Aspects to consider in the production process Natural depiction of the characters' movements Need to shift the audience's viewing angle
Character designer and the 3D worker needs to communicate well in order to produce dense content