Hello this is Juyoung Lim of Westworld
Today we will learn about VFX pipeline and
how VFX pipeline is changing
using real time game engines
You will be able to enhance your understanding
of usual VFX pipeline and
you will learn how new content is made
using real time engine
The diagram you see is a usual VFX pipeline
Pre-production, production,
and post-production are the stages
and each stages have the preparation step for the filming
and the real work step after the filming
VFX engages in all steps from pre-production,
where preparation for the filming is done,
to the filming, and the post-production based on
the filmed data
and we provide solution to those
Understanding VFX pipeline
Now, I will talk about what are done in
each stages of VFX pipeline
and how the procedures are broken into smaller steps
in detail
First, the most important first step in VFX pipeline is
as you can see, photogrammetry
and scanning
This step refers to where you scan
the needed object or environment before or during the filming
and saving them as data
The filmed images will be saved in 2D
and later on, some kind of object or environment
will be overlayed on that 2D image
In this step
the props or environment during the filming
or the information about the location become the most important data
Without these data
you will have to simply guess
how it was filmed, what lighting was used,
and what objects were used
just looking at the images when creating the final plate
This way, the final plate will
turn out awkward
or the balance of the scene
will be broken
So before getting into filming,
what will be filmed, what will be scanned,
and how they'll be turned into data are determined
and once you get into the actual filming, the database is formed
by 3D scanning each data,
using lidar to scan a space,
or using drones to capture them
encompassing every data, prop, and environment
used in the filming process
This process is scanning or photogrammetry,
the data preparation process
Once the filming is done like this
and the post-production begins
the first this is preparing the asset
Asset doesn't refer to something complicated
It means the parts used in projects,
each ones of the used parts
It could be the character
or could be a specific object
What comes first to prepare this asset is
modelling
Modelling process is used when making an actual character
or environments or props
There are many ways like using certain software to carve
or connect rectangles like hard surface
or use photogrammetry,
based on photographs
Fundamentally, the modelling process is the basic step
to make 3D asset
Once the modelling is done
what follows as the second step is rigging
You might have heard of it
and there might be people who have already tried doing it
Simply a bowl or a glass
might not need this rigging
Those, when holding up,
like this remote controller,
how it moves is easy to tell
It's just lifting it up and down
so you only need to produce how the
world axis moves
but humans or animals
and cars, hellicopters and such
have parts moving organically
and there must be a way to control this movement
All these are done in the
rigging task
Rigging basically is
making the skeleton for a 3D data
and preparing to make movements
based on this skeleton
Not only just making the skeleton
but once the skeleton is made and the 3D data is able to move
the 3D data can rip
or stretch
The amount it stretches
or making muscles to create the feel of muscles
or when the rigging skeleton moves, the hair,
or clothes moving along with it
These are all in the rigging part
Once this movable asset is ready
the animation process takes place
I think a lot of you would know
about this animation part
This is the part where the movement is created
As you can see you can create the movements of dragons
or like in Avatar, using motion capture
you can make character movements as well
Animation, using the key frame method
by creating each frame and each rigging control
to create movement
or motion capture used in virtual production
or data captured from facial capture
can be used as a base of animation as well
Animation is the part where the most labor
and time is required
It can't be assured to be perfect just because there is data
but the intention of the director
or the way the content is made
can exaggerate or reduce it
Through these process, animation is done
Like this, once the character
movement animation is ready
now we can say that we get into
the VFX specialized area
Match moving or lay-out
becoming possible is the basis and the reason
of movie VFX and drama VFX's development
Match moving is a technology creating 3D data
from 2D images filmed before
Especially, among the 3D datas,
it is the technology creating the camera
In movies or drama series
2D images are captured through cameras
and this 2D images have objects or specific effects added
through the VFX process
To do this, where the camera was filming
and the space the camera was filming
must be reproduced as a 3D space
to put the created objects, environments, and effects
exactly where you want them to be
This whole process is match move
Match move is the process of selecting a certain area from the image plate
and making the 3D camera out of it, from there
Through this process,
even when there are lots of movement in the camera
3D space can be created
and once this 3D space is created
the objects that fit into that space
can be finally added with the data obtained
So match move artists put more focus on gaining the
accurate data rather than on artistic aspects
when working
And once a character is made through modelling
and it is able to move after the rigging process
to give this character realisticity
the process of texturing and look development takes place
As you can see, real characters
in their 3D mesh for with just grey
cannot be realistic
It needs new texture that reacts to light
and making these textures
and drawing these textures is texturing
and with this texture,
making the texture react to lighting
is look development
This look development uses photo reality
to created to most realistic feel
Of course, according to the occasion,
it might create a cartoon style or a little faded
look in the texture
but usually, the work is done
focusing on how it would look
if it existed in real life
The most important part of this process is
choosing the right rendering engine
because depending on what rendering engine is used
the look can
turn out
completely different
Next is lighting
You might confuse lighting with
look development or texturing
but there are different
Previously mentioned look development or texturing
are process for a single asset
while lighting process is
done on the given environment
If scene A is filmed
and you are to add an object into that scene
the object must be rendered
in the same lighting as scene A's environment
What this means is if you filmed a dark scene
the object being added in
must also be rendered with a dark lighting
to make it look like it's in the same environment
So lighting artist
uses the site's data the most
The filmed data, the weather and environment of that day,
the 360 image of that day, and other references
are used to create identical lighting
with the filmed environment
But rederers cannot perfectly simulate
the lighting in real life
So various tricks
or various methods
are used to make the most similar lighting
in the lighting process
In movies and drama series', unlike full 3D animations,
add objects, environments, or effects to
filmed real life images
So there needs to be a process where
unnecessary parts and the most important parts
are separated from each of the filmed images
This is called rotoscoping
Usually we say we get a roto
and what this means is
as you can see there are many objects filmed
and for each objects
some of them might need to be deleted
and some might need to be moved
or a 3D object related to another one
might need to be added in
So to separate each objects
you mask each object on every frame
which is rotoscoping
This rotoscoping is in a way the basis of
2D compositing
so depending on how the rotoscoping is done
when the compositing is done
the quality greatly differs
So rotoscoping also needs a lot of time and effort
There is also element shots
Usually the characters or objects that we make are
standardized 3D data
Certain object moving
or arms and legs of characters or animals moving
or a creature moving are rigging based
and bone or muscle movement
simulation
But element shots
are not like this and unstructured
like fire, water, wind, according to the scene or shot
It refers to generating objects that
continuously change
So it can be made by filming
or it can be made with
simulation tools that we call FX
Smoke, fire, water, wind, and such are
made in certain areas like this
and added into scenes
Lastly, I will discuss about
compositing
You could say that we've been doing all this
to prepare for compositing
3D character, 3D environment, FX,
and lighting, look development, all these steps
are in fact a compositing of 2D plates
to create one last rendering,
the each elements prepared for that
And with these elements
the final compositing is done
to create the final plate
So compositing artists think about how each
elements will be composited and
how they should be composited and connected
to create a natural image
and how each elements should be blended to look
integrated in one scene and not separated from each other
So depending on the compositing quality
even if the prepared single asset
or single environment has high quality
the final quality is determined by
the composition
so this compositing process is a very important step
Lastly I will talk about rendering
This rendering process does not belong to a certain stage
It's rather a term used in all steps in VFX
from asset or environment, to 2D compositing
rather than a task
Rendering refers to the whole process of
simulating light and creating a final image
It could simply render a grey state
to create a grey video
or render something that only has textures
and create how the light reacts
This whole process is called redering
and it requires time and resource
The resource needed for rendering could be CPU
or GPU
And according to the quality you want for each frame
it could take a lot of time
but it could also take only a short while
By calculating all these
the final quality is determined
so it would help you a lot
to know how this rendering process is done
and what steps it takes
It is impossible to not mention USD
when talking about VFX pipeline these days
so let me briefly touch on it
USD is a open file format
or a file framework created by Pixar
Though it's not completely accurate to call it just a file format
because USD can extend many functions
and share those extended functions
Not only saving files but you can save,
share, and edit all data
of a scene
Before, the reason why sharing and working together
on 3D data and 3D asset was
because there was no standardized format and each file format
had different functions available
but after this USD emerged
it is treated like the standard of the industry so that production is based on USD
and files are exchanged to fit into USD
meaning that different tools, different rederers,
and different software did not matter
and contents of the same quality were possible to create
Another reason why USD is important is
also on virtual production that we'll talk about later on
virtual production also uses 3D data
and to make use of 3D data
using USD format data
makes it possible for it to be used in other virtual production
without any other conversion process
So along with VFX pipeline
if you study USD
it will help you greatly
I would also like to tell you about
management in VFX pipeline
The process I told you previously
can be implemented on certain shots as a whole
or only certain parts of it as well
There are some shots where without any other process
only compositing is needed
But in one movie, there are 2,000 cuts
and in one drama series, there are 8 to 10 thousand cuts
To work with these
there will be
80 thousand, or more than 12 thousand tasks
and it is bound to require a lot of labor in the process of
collecting, delivering, and confirming the data from each tasks
So management is the most important element
If VFX artists or those working in the VFX industry
do not have proper understanding of this management
they cannot have the full image of how they should
deliver and confirm the data they worked on
and how the final output will be
turn out to be
So knowing what management tool you are using is
and what it's composed of
and how it's related to the tools and data
you are working with
will help you greatly
Conditions for linking VFX pipeline and unreal engine
Until now, I talked about usual VFX pipeline
and from now on
I will talk about how real time engine, game engine
integrated into VFX pipeline
and what change it brought
The first example I will explain would be
using game engine
to create a VFX crowd
First, if you use a game engine
you can check the modelling, rigging and
animation process in real time
Before, doing this type of cooperative work
would require each piece of data to be modeled,
textured, and animated
using differing tools or by having to convert the data
every time, but with game engine
various data can be brought to one project
where the animation can be checked in real time or
edited when making characters
Before, this cooperative work
required a lot of time to check each object's animation
since it would require
high resolution rendering
but using game engine, a unified system
let's you check animation quickly and easily
And another strength that game engine
has is AI
Here, AI does not refer to LLM like generative AI
or GPT that are famous these days, but AI in games
AI in games means designating
how NPCs move
For example, where the character should go
when I throw a first at them
or what action to show when seeing what character
These general logic are all called AI
This AI can be made and edited in real time
through visual scripting or source code
For video tools that were used before
these AI would be directly related to animation
which means that generating animation and effects between subjects
and the interaction
needed a lot of steps
but with game engine
relationships and actions between subjects are
as you are seeing, made quicker in production
and also in checking with visual scripting
Each characters made like this
must be placed to fit the scene
Here we can see another advantage that
game engine has
The landscape data you are seeing is a 3D dummy data
made from an actual filmed plate
Dummy data refers to a data
that is turned into guide form 3D of a certain area
when making 3D camera in the match moving process
On this dummy data, how the
AI crowd is spawned is determined
and then from that area,
whether the density must be higher or lower
is able to be determined
And among the areas
where crowd shouldn't be on can
also be determined with dummy data
and that way, where reactivity like
objects or buildings are located don't generate data
and areas only without reactivity will
generate crowds
With all this process, AI characters, and
using match move data made in original shots
continuous and short time simulation
can be checked real time
Like this, AI characters and located crowd
can be live composited in the game engine
Before, to check the composition of these data
you would have had to render
a generated 3D crowd data to a 2D image
and then bring this rendered image and the original plate
to the compositing tool
but with game engine
the characters are located in real time
and how these characters are shown in
2D image plate can be composited live
Of course it cannot reach the optimal compositing quality
but checking the basic locating and depth
allows easier editing
and confirming
Lastly, this was one of the challenges
The images made like this in the game engine
are then extracted of data for the actual compositing
But the limit that game engine still has is
that it cannot perfectly separate the many
passes and layers of images used in the
original compositing pipeline
So various passes provided by game engines
would be extracted and then these passes
would be used to find the desired pass
using many color controlling software
As you can see
what you are seeing is
the passes about 3D crowd data
And these passes are
used to make the final passes
with adapted titles and compositing
But this as well, as game engine technology developed,
with updates, is
expected to create functions that can
customize and apply on all pass areas of
general VFX pipelines
From the passes that are made like this, the
selected final 5 passes are used in the final compositing process
This final compositing process
is not done in the game engine
but in the software that is
most commonly used in VFX
And the advantage of using the software
most common in VFX is
that it does not interrupt the structure that
VFX pipeline originally had
and that if only the data source that is delivered at last is modified
the whole process of compositing can be structured
So it uses the flexibility of pipeline
and the advantages that the original pipeline has
and also cover for
its disadvantages
The video you just saw is
a final video that uses a crowd
that used a game engine and a filmed image plate
Comparing it to a crowd video we worked on before
it is an example that is not too far off
and even now, these crowd characters,
we use two tracks while working on them
Lastly the second example is
a case where the original pipeline
was maintained while integrating a game engine
into it
The title consists of specular workflow and unreal engine
It is the effort to reduce
the gap between original commercial renderer,
that is, renderer used in VFX, and
the rendering system that
game engine has
First, it can differ from
company to company but
4 textures are most commonly used
Diffuse, specular, normal, roughness
These 4 texture layers are used
to make 3D data
What you're seeing is the images of the
4 textures we used in a drone that we made
and the 4 textures made like this
are materialized using a method
called specular workflow
Specualr workflow works
the complete opposite way to how a metallic workflow does
Physical base rendering method
can be considered one of two methods
Metallic and specular are different
first, how they differently
express metallicity
Metallic workflow expresses metallicity based on
the presence of the value of a metal, that is,
whether it actually shows metallicity
while specular workflow
expresses metallicity according to
how much the albedo(reflection value) is
The problem is that game engine that we use
use metallic workflow
and normal commercial renderers use
specular workflow
So if one brings a texture
directly to a game engine
the metallic would be expressed but the specular might not be
or the specular would be expressed but the metallic might not be
So as you can see
to solve this problem
master material was created
Master material for specular workflow
If you took game engine courses
you might know about materials
but to give a brief explanation
It is a shading system used in game engine
Using this shading system, it can make
a shader with various functions node-based
Through the production of this master material
we made a master material that can convert specular material
and metallic material, creating the same look without change in texture
but only converting the shading
And with this master material
we were able to materialize specular workflow
on a game engine without texture conversion
If we used the original game engine basic material
the specular would not have been expressed at all
but using the master material that we made
allowed the specular map
to express specular as well
This process is
being shared on a market place
So if you are interested
you can use the Westworld Plugin
to use it while taking the
game engine courses
This video shows how a Westworld Plugin works
What's important here is the USD pipeline that we made,
the metallic workflow used in unreal engine,
and specular workflow are integrated
allowing us to create a
final 3D asset that has the same look
with just a single asset on a
game engine
and this allows asset, environment, or creatures
created in VFX to be used right away
in game engine without particular conversions
and this in turn allows
the increase of assets that can be used
in a virtual project
Third example I will introduce is
live virtual camera and animation production process
This isn't anything special but I wanted to tell you about
real time pre-viz, pre-visualization and how game engine
is used in there
You might have heard of pre-visualization
It is also called pre-viz
Pre-viz is the process of creating a 2D or 3D image of
a certain scene or sequence to show the director
and determine the method of filming
or the filming angles
Before, these process use tools like
DCC, Maya, or Blender
to reduce the VFX process
and do rough modelling, rough rigging,
and rough animation
But now with game engine, motion capture,
and virtual camera technology
it can be done in real time
This helped to enhance
production efficiency a lot
and not only pre-visualizing certain scenes
but doing that to all sequences,
that is, every sequence of a movie or a drama series
can be pre-visualized
The video you're seeing shows
a motion captured movement of the actor
and the red camera movement
is made using real time camera tracking technology
It wasn't used in a project
We had made it for a test
Using the same method, with the name of real time pre-viz
one can film
and use the camera image in the bottom
to make an actual video
and form pre-viz data
As you can see, if these animations were
filmed by real people
and even the camera animation was captured by people
the process would have taken a lot of time
but with motion capture, performance capture of virtual production technology, and
camera tracking technology, it can be
done in a short period of time
Virtual production, especially game engine
has a lot of advantages especially in
this camera animation
Camera animation before
would take place where there are
no actual lighting
so it's hard to check what feel it would give off
and how it would be filming
but in game engines, not only real time lighting effect
but also lens effect, the actual
physical effect can also be simulated
and making a track or moving itself
can be made like assembling something
as if in a game, so camera animation can be
produced much quicker
Not just what's behind
but certain SF scenes
also can be produced
using camera animation process
What you're seeing now is an actual camera's
movement flow with a simulation data
You can see here that physically impossible
camera animation can be done in game engines
This cannot be used in actual filming
but once it's connected with
full 3D characters and full 3D environments
the scene will be possible to generate
using this data
So if camera animation
and motion capture technology is actively used,
using this real time pre-viz technology,
angles or scenes that were unheard of
will be possible to make very quickly
Lastly, one thing that
was challenging was that these technological elements
like moving with the camera
or capturing the camera animation
can be difficult for directors or filming directors
that have less experience with game engines
But with this project
we did a process that let them control the camera
as if they were playing a game
As you can see, the movement, recording, editing,
all functions of a camera were connected to a game console
and the director could
create the environment they want
without specific aid
If this was done
through communication
the director would have to tell the animator
and the animator, after doing their part once given the order,
would have to render the final data and again get it confirmed
This would have taken at least a few days and that would
cost a lot of time and money
But if real time rendering is possible
and real time interactive system is
possible, one can create
what is desired by controlling it oneself
which makes the production efficient
What you see now is a final pre-viz video using
real time pre-viz, camera tracking,
and movement generation
through xbox control
I have talked about
VFX pipeline and how it is composed
looking at each stages
And also how virtual production technology
is used within the VFX pipeline
and how it's integrated
As you continue to learn virtual production technology and realistic content technology
and create videos or animations
I hope this can give you an idea
of how you can make use of these
Thank you
Understanding VFX pipeline VFX pipeline
Conditions of linking VFX pipeline and unreal engnie VFX crowd production using unreal engine Character setting AI controller and character blueprint
Crowd positioning using PCG Crowd positioning using PCG and real time composition using composure
Specular workflow and unreal engine Westworld texture construction Metallic workflow that expresses texture according to metallic value
Specular workflow that expresses texture according to the albedo(reflection value) Master material for specular workflow
Live virtual camera animation production process