Hello, this is Ju Yeong Lim, the director of Westworld technology research
In this GCC realistic content curriculum
I will tell you about virtual production during the fourth session
Among virtual productions
I would like to talk about movies, dramas, or XR content
First, in the first session, I would like to introduce the concept of virtual production
And provide a general overview of how it is used
Virtual Production's Concept, Entertainment Industry, and Animation Use Cases
Some of you may have heard of virtual production
While others may be unfamiliar
And even those who have heard of virtual production
May think it is difficult to approach because it is a technical topic
In reality, there are indeed many technical elements among the various elements
That make up virtual production
However, it seems that the most important factor is not simply operating with technology
But what kind of content will be created with this technology
So, the virtual production
Rather than focusing on the word virtual production
This is a part of the production process
Virtual Production Concept 1 And is a set of technologies and used to create content
It involves not only technology
But also filming, post-production, lighting, art, and everything else maintained before
Virtual Production Concept 2 So that you can think of it as a fusion of existing and new virtual technology
First, you are looking at the picture you see now
About the virtual production pipeline
Unlike the existing VFX pipeline
Virtual production is organically related
And content is produced as it continues to develop
Suppose the existing VFX pipeline or production method
Goes through a series of processes from shooting to post-production
Virtual production is
A series of processes from shooting to producing final content that are organically connected
And content is continuously updated
In XR-related content, animation, drama, and movie content
This virtual production technology is used mainly
I will explain how it is used in each field
First, let me tell you about the use cases of virtual production in the entertainment industry
I think virtual production is probably actively used in the entertainment industry
Considering the number of
Films or dramas produced and the production period
Even if actual virtual production technology is used
It takes time for viewers or content enjoyers to consume them
Through entertainment performances or YouTube content, these entertainments
A market seems to have been formed where XR content can be created and consumed immediately
As you can see, auditions using digital characters
Performances using characters such as digital idols
And YouTube contents are continuously being produced
Virtual production is being used for this production technology
First is the XR extended performance
So, to know about examples of virtual production being used in XR
You need some knowledge of motion capture
This motion capture is probably
So deep that you must create a separate class to take it
But generally speaking
Motion capture refers to converting human or animal movements
Into digital data
There can be many ways
As you can see, using an optical camera
Or a gyro sensor
Such as the gyro sensor contained in a cell phone
That uses an axis sensor receiving motion
Or by using a video
Using only bone data of a person or animal in a video
To get motion capture, etc.
Using this motion capture data
Animations that had to be done manually by humans
Can be alleviated by 70 to 80%
So it makes quick production possible
This quick production is
Being used to create content quickly
For performances or YouTube content
What you are seeing now is a Real-Time Motion Capture
A function that changes actual movements
Into 3D data right on the spot
The most important thing here is that
I think it would be good to know the term Retargeting
In fact
The dancing actor in the middle
And the 3D characters moving on either side
May appear to be moving in unison
However, if the ratio of the prepared 3D character
Or the actual actor acting is incorrect
Issues arise, such as
The actual movement is torn or appears strange
These can be corrected through a function called Retargeting
Rather than a difficult technology
It would be better to think of retargeting
As a technology that actually reattaches the position of each bone to a new bone
In these XR contents entertainment
The method of producing content using a game engine
Has the advantage of allowing one content to be used in many different forms
As you can see, creating a digital idol and using it not only for performances
But also for creating interactive content that can communicate with the digital idol
Making a game and selling it
Or making and selling another movie also
Utilizes the 3D data that was previously created.
Because it is an area where it is fully possible
It seems that the method of producing through virtual production is emerging
In entertainment, virtual production technology is
Also changing the way entertainment show is produced
The video you are watching is a show made in Hollywood called Alter Ego
The show used real-time motion capture
Performance capture
AR, XR compositing
To create content that features real-life interactions with digital characters
What you see is content called Ava Dream, which was made in Korea
In the same way as Alter Ego, this content also
Used motion capture, facial capture, and real-time AR, XR compositing technology
To produce new entertainment content that had never been seen before
In producing all of these entertainment contents
Motion capture, camera tracking
And AR/XR compositing technologies used in virtual production ere utilized
Korean virtual idol Plave
In the case of the real-time idol performance you see
As the Ava Dream and Alter Ego seen earlier
By using motion capture, facial capture, and real-time AR compositing technology
Making it possible to produce content
Featuring these idols performing in actual live broadcasts
This type of idol content production
Continues to be attempted in many ways
Because it can lead to a constant influx of fans and form a consumer market
Korean virtual idol MAVE: A bit different from the Plave we saw earlier
You are seeing a video of a live-action digital character being created
And an actual performance being performed
Of course, in existing entertainment
XR content production is pursuing faster speed or more efficient production methods
But by using virtual production technology
By creating digital twin characters seen in movies and dramas
I wanted to show that it is possible to create high-quality digital content
Another thing you need to know about virtual production is
The concept of XR performance expansion
The videos you see were
Shot on a green screen or LED wall
As you can see, the LED wall or green screen was
Used only in the visible area, not the entire area
However, the actual environment shown is
A video created so that the performance can be performed in an expanded form that covers the entire area
These technologies are being used in music video production or entertainment
Emerging as a new video production method that has never been seen before
So, as you can see, this XR content and performance expansion technology
Utilizes a method of transmitting existing sources, produced 3D videos
And people or objects that are actually filmed
Through real-time compositing
The technologies used here are
Green screen compositing, motion tracking using camera tracking
AR compositing, and XR compositing
These technologies are being put to use
What you see is a diagram of the XR performance expansion
As you can see, within an actual LED wall or green screen
Live shooting is proceeded
Adding another new source to the LED wall and green screen
And transmitting a composite image
In entertainment, virtual production technology is
Emerging as a new content creation method for
Performances or idols you have seen before
And for commerce, interactive, and realistic videos
You see a video of an e-commerce transaction
Using an actual LED wall and XR technology
The second thing you see is
Not only a video filmed on the existing LED wall
But also a video made using XR technology
So that you can watch the show as if
The actual idols were performing within the space rather than in front of the LED wall
This XR virtual production technology
The virtual production technology on the XR stage
Enables live communication between the audience and actual content creators
The content we are creating is
Not rendered and played back statically
But can continuously interact interactively in real-time
Whether it is lighting or audience motion
Because continuous communication is possible with any specific field
Creating an environment that can provide a more interactive
And different experiences during actual performances are possible
So far
We have looked at virtual production utilization technologies in entertainment
Generally called the media entertainment field
From now on, I will talk about examples of
How virtual production is used in the field of animation
What you are seeing is an animation called Yumi's Cells
This animation is 100% rendered using a game engine
Animations such as Yumi's Cells, Armored Saurus, and Willbengers are
Animations produced using game engine real-time rendering technology
Existing animation overlaps several steps, starting from a storyboard, creating a character
Planting a skeleton on the character
Animation
Then working on textures and lighting to suit the animation
And finally compositing
Each of these steps requires enormous resources
And a lot of time
Confirmation was required for each step
And it is impossible to know 100% perfectly how it will look
Until you go through the actual final rendering process and see the image
However, real-time rendering engine
Especially animation production using game engines
Has the advantage of being able to check all of these processes at once
Although it is impossible to check the final lighting with just one button
It has the advantage of checking the output immediately within the actual working time
This advantage
Allows you to get faster feedback
Which will help you make faster revisions and produce the final product
In animation production
When using a game engine and motion capture or performance capture
It can be implemented in real-time
To create the picture desired by the creator and director
And confirm it to the final quality
The diagram you see is a diagram of the entire process of
Implementing animation through motion capture in real-time
The video you are watching is
About something called performance capture or facial capture
As you can see, on real actors
To Acting actors, headgear is put on
And the animation is
Created live as they act in the headgear
Even if we had previously received such animation
Especially facial data
Many processes were required
To import this facial data, process it, and proceed to actual lighting
However, we can now see that it is possible
To confirm it in a form equivalent to the final quality as soon as it is filmed on-site
This real-time animation creation technology is
Not only used in animation
But can also be used in live-action filming
What you are seeing is a scene from Jingle Jangle, one of Netflix's shows
Where actual footage and 3D animation characters were
Live composited through performance capture
Through the live composite data
The final result was produced
As you can see, in the past
The actual animated 3D characters and the live-action
Would have been filmed separately and later composited
However, if you utilize live virtual production technology
By viewing live-action shooting and live characters simultaneously
You can achieve faster confirmation and production efficiency
Animation production using virtual production has
Also been attempted in Korea
If you look at the work called Armored Saurus, which aired in 2022
Real characters, 3D environments
And all moving characters filmed on a green screen were
Composited live on-site using a game engine
And were finally aired after post-processing
Without these technologies
To shoot green screens
Create real environments
And dress up characters
It would have taken a long time and a lot of money
But by using these virtual production technologies
I think we can create content of decent quality in a short period
Almost real
Of course, we cannot say 100% that it is of real-life quality
You can see that it is possible to produce 3D animation equivalent to live-action
Through this Armored Saurus case
The diagram you see
Shows a pipeline that uses a real-time game engine and real-time motion capture
To produce photorealistic composite animation
As in the Armored Saurus example above
The environment and characters prepared for live-action
are rendered in real-time at the live-action shooting site
And the final result, the final plate, is filmed through compositing
This method of producing animation using virtual production
Has the advantage of
Quickly producing things that were previously difficult to attempt in animation
Or were not cost-effective
If you were to produce a CF using that cartoon rendering method
You would have to go through the entire animation process
And many processes, including actual lighting, would be required
Making it difficult to achieve the CF production cost
However, by using game engines, real-time rendering technology
And motion capture technology
In short time, cartoon rendering
Or advertisements with an animation production became possible
Drama and Movie Content Use Case
So far, we have talked about virtual production
In media entertainment or animation
Now we will talk about dramas and movies
In general, I will talk about
Virtual production, an area combined with live action
If virtual production technology is responsible
For the final output in XR content or animation content
In drama or movie content
It would be good to think of it as being used in a form that helps with pre-visualization
or filming method production
Such as ICVFX
There are ways to produce actual final plates
Still, it would be good to know that
Not only those methods but also each technology of virtual production is
Used throughout the entire movie or drama production process
The Silent Sea
You are watching a video of
An actual final plate filmed using an LED wall in a project called The Silent Sea
As I mentioned
This is an example of
How virtual production technology can utilize the final plate
In addition to these examples
Through the various scenarios explained later
You will see how virtual production is used
In movies and dramas
You are looking at
A pipeline where virtual production is used in movie and drama content
You can think of it as similar to animation or virtual production
Pipeline and picture above
Virtual production technology is being used
In all areas of content creation, from the pre-production, production
And post-production stages
We wanted to show
How this leveraged virtual production streamlines how content is created in the real world
To explain briefly
In reality
Existing scenarios were produced, and before making a video of it
With the concept of a storyboard
A specific image, or a keyframe image
By using current virtual production technology
All of these images can be created
Into 3D images called actual sequence images
And actual shooting is performed using this image data
And the data used in the shooting
And the image 3D data prepared previously are combined
It explains the process of creating the final output using
〈The Silent Sea〉
In the video you saw earlier
By using virtual production
Previs and actual ICVFX
And post-production throughout the entire process
You saw the show reel of The Silent Sea
From now on, I will explain what technologies were used in The Silent Sea
And how they are used
First, as explained earlier, virtual production
Allows new content to be created in the pre-production stage
Previously, based on the storyboard concept
Storyboards were made into scenes
Or only important scenes were made in full 3D
However, by utilizing this virtual production technology
Especially technology such as motion capture and real-time camera linkage
The entire area of ​​the storyboard can be created in full 3D
The director can change the desired angle anytime
And create a new shooting plan
You are seeing
A real-time previs test video of the Sweet Home project filmed in 2020
Sweet Home Real-time Previs
Previously, when filming such a creature
There was an issue of having to film on a blank green mat or a blank field
And then composite the creature later
Then, on set, actors or directors
Had to look into space and give directions
Or they had no choice but to film based solely on their imagination
However, using real-time compositing and real-time previs technology of virtual production
You can create an environment
Where you can film while checking the character's movement and scale in real time
And because the data captured in this way
And the 3D data synthesized later in post-production are maintained consistently
The actual output quality can also be guaranteed to be high
The video you are seeing is called real-time previs
Before the actual movie or drama is filmed
So you are watching on-site footage that uses motion capture and camera tracking technology
To create a specific scene
When you look at the actual picture
It seems like a person is simply moving and filming it with an iPad
But all the data is being collected from that single action
And 3D environments and characters are being created in real-time through this data
By using this technology
As mentioned earlier
The angles or imagined actions of the director
Can be expressed not only through drawings or writing
But also through moving images
Which not only improves the efficiency of the production method
But also allows for more possibilities during production
Sweet Home Simulation Camera
In the video you saw earlier
The simulation camera is generally referred to as a Simul Cam
And the term Ncam is also used as a specific term
As you can see, in Sweet Home season 1
This is a video where you can see how it was filmed using the simulation camera technology
And how the filmed data is used
Until the final output
Lastly, virtual production
Among virtual production technologies in movies and dramas
I think the most talked-about
And most useful technologies are
In-camera VFX and 2D Driving Comp
You can think of one area of In-Camera VFX as 2D Driving Comp
The elements that make up in-camera VFX are
Divided depending on whether they are composed of a full 3D environment
A 2D image plate
Or a combination of the two types
To create a 2.5D environment
That combines a full 3D environment and a 2D environment
You can think of it as an In-Camera VFX
That encompasses all those areas
In-camera VFX is a method of erecting an LED wall
And placing the environment or image that requires actual filming on the LED wall and filming
This method is not
A simple background or a stationary background called hole
Because it is an environment that can be interactively interacted with in real-time
Because it helps with immersion and acting on set
And it is not just a light-emitting object
It also performs the role of lighting
This ICVFX technology was first introduced
In 2018 by The Mandalorian
The Mandalorian is a spin-off of Star Wars
In which the environment changes continuously while traveling to various parts of the universe
And the story takes place within this environment
This isn't making a two-hour movie
It's making an episodic drama
Therefore, if you build all sets or film on location
In a constantly changing environment
Production costs are bound to increase exponentially
So, for the first time, they built an LED wall
And filmed by projecting various space environments onto the LED wall
Through this, it became possible to freely move between various environments in one set
And produce high-quality content while minimizing costs
Through this case in Korea
There have been cases of attempting this type of ICVFX
What you see is ICVFX on the moon, which was first attempted in The Silent Sea
The difference between domestic and overseas cases is that
In the domestic case
The actual location area
was not carefully adjusted from the pre-production stage
So that usability may be slightly lower
However, when it was actually filmed
It was a project whose potential was fully confirmed
I can tell you that the most used one is 2D Driving Comp
I put 2D Driving Comp separately under ICVFX
Because it is the most efficient method compared to existing shooting methods
So I wanted to introduce it separately
So, I think I will split the episode and talk about 2D Driving Comp in more detail
Simply put
Any scene in a movie or drama
That takes place in a car or any other vehicle is included
Whether it's the future, the past
Or, of course, not if it's a much past
In contemporary, modern, or future works
Many conversations
Or actions take place on something
Previously, all these scenes were filmed on a painted mat
It was synthesized later
Because of the later compositing
It was not easy to match lighting issues
Actors' acting, or interactions
All objects are made of glass
And if we tried to composite them
By implementing reflections on the glass or moving environments outside
There were problems with time and quality
However, the use of ICVFX
Seems to be optimized
For filming these vehicle scenes, especially dialogue or action scenes
Although it may vary depending on the size or size of the LED wall
If you build a size that suits the vehicle or a specific vehicle
And film it through prepared or created video
You can feel immersed in the space
However, it seems to have a strong advantage
In terms of the characteristics of the window glass and its ability to express reflection
The video you are watching is
A 2D Driving Comp used in Netflix's MANK
The reason why I showed this 2D Driving Comp separately is
Because this work was filmed in color
This work was later converted to black and white
So, in certain areas, some characters appear in color
While others change to black and white
If you look at it now, when viewed in actual color
The 3D environment may look awkward
However, when converted to black and white
You can see that it blends into the scene exactly like this without any sense of incongruity
I wanted to show that this virtual production technology
Depending on how the idea is created or used
Not only reduces production costs
But also has many possibilities in terms of usability
2D Driving Comp is not just about vehicles
It can also be a train, a helicopter, or any space object
As you can see, filming on a train is not easy
Not only is the space of the train small
But it is impossible to know where the train is running, what scene to film
When to pass through the tunnel
Doing the action is impossible
And it is difficult to film
Because the environment can change from time to time as a person operates the train
But by using this train set and LED wall
Exterior background you want
The timing you want
And the perspective you want can be made and filmed
So, to create train scenes
Green mats and sets were built
But now it is possible to create
Content of the same quality through ICVFX technology and sets
Among the ICVFX cases, the most successful one I would like to introduce is
A project called Netflix 1899
This 1899 work is set
During the Enlightenment period, that is, from the late 1800s to the early 1900s
And most of the costumes and environments at this time are
Set, and it takes a lot of money and time to build them
Since most of the action takes place on a ship
The background of the ship consists of water, sky, clouds, etc.
Combining these with ICVFX and art set technology
Through minimal set production and maximum environmental 3D production
Efficient filming was enabled
Such Driving Comp filming continues to take place in domestic as well
You see the Driving Comp shot used in The Queen of Tears
If the two actors filmed this conversation scene
Outside in the rain
It would not be easy to match the shooting schedule and rainy day
Ad it would not be easy to continue filming on a rainy day
However, through the LED wall, captured sources, and ICVFX technology
While continuing to retake
We continued filming until we captured the desired emotion of the actor
As a similar example, in Season 2 of Netflix's DP
You are looking at the Driving Comp shot filmed
As you can see, the first attempt was made by
Putting a truck and a car in front of the LED wall simultaneously
And filming while the two cars were interacting
As you can see, the size of the LED wall is important
But, depending on your shooting method
I wanted to show you that you can shoot the scene you want with ICVFX
And it's the same project
You are watching a scene using the train and LED wall used in DP
This scene also involved a lot of action by real actors
And with so many changes in camera angles
It was impossible to film it on a train
Although the train was got
If all the exterior windows of the train were filmed and composited together
The production cost would inevitably rise significantly
However, by using this LED wall technology and ICVFX technology
We could film all episodes in a short period of time, in two days
And produce the final plate immediately without any special post-processing
So far, I have told you what kind of technology virtual production is and how it is used
If you think of virtual production not simply as a technology
But as a way to create content
You can use it in many ways to create new content in the future
Or in any project you use, I think it has potential
I hope this helps you a lot, thank you
The Concept of Virtual Production and Examples of Use in the Entertainment Industry and Animation Virtual Production A set of technologies used to create content
Producing content through organic connection and fusion of existing and virtual technologies Motion capture, performance capture, facial capture, real-time AR/XR compositing technology, etc.
Entertainment Industry Use Cases Virtual idol content: PLAVE, MAVE: Entertainment content: AVA Dream, The Golden Age of Subcharacters, Avatar Singer
LED Wall + performance expansion content Commerce content LED Wall + Interactive Concert
Animation Use Cases Real-time rendering that allows you to check the output while working provides quick feedback and final results
Autonomous confirmation, flexible pipeline, high-quality results in a short period of time Works: Yumi's Cells, Armored Saurus, WillBengers
Drama and Movie Content Use Example Real-time Previs The entire storyboard area can be created in FULL 3D and visualized with data A production method with real-time feedback, improved quality, increased efficiency, flexibility, and creativity
In-Camera VFX Creating the environment necessary for the LED Wall and filming Interactive real-time interactive backgrounds
2D Driving Comp It is a type of in-camera VFX technology and has the highest utilization and usefulness Optimized for dialogue scenes and action scenes inside the car