Hello, this is Juyoung Lim from Westworld
In this lecture, we will take a look into in-camera VFX projects
and form an understanding on ICVFX
and see how they are used
Through this lecture
I hope you can learn
what technologies are used and what is important
to use in-camera VFX and also how it can be implemented
in actual filming circumstances
Composition of ICVDX
First, I will tell you about the preparation process
of in-camera VFX
ICVFX start from the pre-production stage
and continues all the way
until the end which is the post-production stage
In this table
you can see that contrary to the normal pipeline,
there are environment asset preparation
on the pre-production stage
and this environment asset is quality checked for lighting
and then in the production stage
the whole final plate is produced
and within this plate
the post-production stage takes place
making the final product
To tell you what ICVFX is once again
I will tell you the structural elements
of ICVFX one by one
First is the most important LED volume
LED volume refers to a media wall
made of LED
As you can see
it is a desktop environment that is
separated from the real set or where the actors are
where virtual environment or 2D image is
played with LED
The actual production method is determined
by how the LED wall is designed
The size of the LED wall, how the LED wall is composed,
and whether there is an LED wall on the ceiling
or if there is one on the floor
what pitch the LED wall on
the walls are composed of
if they are in low pitch
and how bright they are
These all affect
how the content is composed and
filmed
So when beginning on the final ICVFX
it is good to think about how to design the LED wall
and if the production takes place where there
is an LED wall
it would be good to
first see what construction the
LED wall there has
The second important element is
in-camera VFX network
This network element
doesn't refer to an actual network
but it's more of a connection
It could be an IP based network
using switch
or it could also be a network
sending video signals or other signals using SDI cable
The diagram you see
gives details on both types mentioned
What's important here is that production using ICVFX and LED wall
is not done with just one device
Rendering engine, media server, image processor, camera tracker
All these devices must move in the
same speed in one network
and must be synced
So if there is a lack of understanding in how this network
is composed, what signal comes from
where and goes to where
there will be many difficulties in the
production process and also
in the content preparation process
So it would be good to start from
understanding where in the in-camera VFX diagram
you are, and what
is important in the area
that you are in charge of
Third is NVIDIA sync
Even if we take out the NVIDIA
I want to tell you once again about sync
Sync here
are two things
First, sync within LED wall
and the sync
between network devices
These are the two
First, the sync within LED wall is
is an LED wall uses a single processor
meaning one rendering machine
or in one media server
this sync might not be a big deal
But if LED wall becomes of larger volume
and many rendering engines
or many media servers
are used to playback it
each rendering engine or media server
must have the same speed and rendering
playback source or 3D environment
to screen it on camera
If not, a certain part of the screen can be ripped off
or a certain artifact can stand out awkardly
and this will affect the production
as it will not be able to use as the final plate
There will be more labor and cost
needed to fix these errors
so to solve these
all elements of the LED
must move
with the same sync
NVIDIA sync is an example of that
These days, not only NVIDIA sync
but also using other methods like SMTP 210
one can achieve the sync method
so I think studying about this
will help you
Net element is multi-user editing
Multi-user editing is
a term used in a certain game engine
But other tools
other media servers or game engines
also provide similar solutions
This solution is
needed to control environments when being on live
or while filming, or to cooperate with
other people
Without this function
it is impossible to edit certain parts or
apply real time changes to
the 3D environment or 2D image plates
that are being played on the LED wall
With this multi-user editing
various devices in a session
can be spread out to everyone in that project
and saved
and load it
which enables controlling the lighting
or camera lenses
or adding specific task
to a certain area
So when doing an ICVFX project
this multi-user editing,
multi-user session server's composition
and how one can access it is
a very important element
Next is perforce server
and I would like to talk about version control system
in general
Version control system is something you might know
if you worked on Github or development tasks
It is a system for managing
source codes
It is a system used to
check all the history
of the development process stacked from
when the development started
and when going back or forth to a certain stage
or editing or recovering something
and only with this system in a game engine
a project can be shared
or update, maintain,
or repair the project
Also, when LED expands in volume
sending data to each rendering machine
or media server also becomes a burden
If a version control system is in place
and each rendering machine or media server is
connected to this version control system
with the switch board you see,
or a similar tool
just a click on a button
can send all data
or apply an update
very conveniently
So if someone asks to choose the most important elements of ICVFX
whether a version control system is
there or not
greatly influences the effeciency
of the preparation stage of projects
and help save resources
Next element is level snapshot
Level snapshot is
an advantage that the former ICVFX
did not have
It is a function that emerged with the new version of
unreal engine, game engine
Level snapshot is a function
that can save all data that is being filmed
at once
What this means is
everything like the movements of objects,
camera movement, lighting effect
in the game engine
can be saved with snapshot
at each moment
and you can also check or go back to it
This way, editing the data after production
or moving to other takes
in the middle of production
is possible
Before, this would have needed extensive note taking
or making a software
to move back and forth
but now, with this snapshot function
just a click an save a certain environment
and going back
is also available with just a click of a button
Next is switch board
Switch board is a software
used to manage various devices
in a network
of a game engine
The reason why you should know about this software is
it not only simply runs the game engine
but also with this software
game engine, motion tracking device, camera tracking device,
all these devices are
available to centralized control
and the multi-user editing mentioned before
or version control system can use this as well
So no matter how many devices there are
Even if the LED volume is huge with 10, 20 rendering machines
or if the media server or camera tracking system
has various sets
If this switch board system, that is,
a system that allows centralized control
launching a project
or updating all data
or editing a certain area is possible
So when it comes to building an ICVFX, especially a 3E ICVFX,
knowing this switch board and its function
and using it will be great
Now I'll talk about UE5
When in-camera VFX was
first made through Mandalorian
it was made based on Unreal 4
With Unreal 5 releasing on 2022
ICVFX's structure also changed a lot
and it has developed to the level where it can be used for actual production
Especially the function nanite
and lumen
allowed the original game engine-ish
or sort of an animation-like quality
to be developed to achieve
a more cinematic look
Also, new things being developed and shared on new game engines
are discovering and improving
areas that were impossible
or inconvenient in ICVFX
Having interest in these
and testing, knowing these itself
is an extremely important element
in developing and using in-camera VFX technology
Another important element is scheduling
What you're seeing now is a
scheduling for a Hollywood show
that uses an in-camera VFX
3 to 4 month of preparations is needed
and building the actual set and pixel
takes another 3 months
With the set and 3D environment made like this
it takes another
month to customize them
With this environment
the process called prelight
where all staffs and directors gather
to film and rehearse
takes at least 3 days
If this scheduling isn't kept
a good quality cannot be achieved
In-camera VFX is taking the steps that take place
in the post-production stage forward
so it is obvious that the first stage gets extended in time
But, with this longer time
more shots and more scenes
can be taken quicker
so the cost that happens in the pre-stage
can surely be of the similar level
of what is saved in the post-stage
When doing an ICVFX project
knowing how the scheduling is
and how I can prepare for it
to make the best quality
within the given schedule
is also one thing you must know
Examples of virtual production
I will show you how the elements
we have just talked about in the example of
'Black Knight' project that we did and
how we used them, how we prepared them,
and what could be improved
Black Knight is a Netflix project released in 2023
I don't think it did too well
so I'm not sure if a lot of you would have seen it
but taking a look into the synopsis
it is an action drama taking place in a desertified city in the near future
So the apocalyptic world
had to be generated
and there were a lot of car action scenes
which made us think that ICVFX could be put into use
so we used the in-camera VFX technology in the production
As you can see, Black Knight had
this scheduling
Scenario was analyzed in the pre-production stage
and with that data
we came up with concepts or key images
to use in the in-camera VFX
And with this key images as the basis
we worked on 3D environment and 3D driving comp data
Then after the rehearsal,
we did the actual filming
It did take the
usual scheduling steps, but
the time span was a bit shorter
than what is recommended
This as also our first project where
we broke the roles into smaller pieces and worked in teams
while doing a virtual production project
As you can see, we divided the roles of
virtual production supervisor and general VFX supervisor
and under the virtual production supervisor
we had a developer that can worker with game engine
and an engineer who can do virtual production
and a team that can research and develop
the technology of virtual production itself
These three teams were in the production
Each teams made environments
or also analyzed what is important
when filming with those environments
and also in the filming
they also looked for methods to
communicate with key staffs
I will tell you about each of the steps
in better detail
The background of Black Knight that we chose
through synopsis analysis was 6 in total
As you can see, floating artificial structures in the nature,
and that structure changing
which shows LED walls
and a huge space called
an air road
and another wall
The black wall
And a lab where experiment subjects lie
We concluded that these 6 backgrounds and
the scenario could make use of LED walls
With these 6 key images
we started working
What we first did was with these key images as the basis
using the storyboard and pre-viz data
we determined which angle
we would film it
As you can see, with the images
given in the storyboard and the pre-viz
we analyzed how to put up the LED walls
and what angle would work out
when we placed the camera how and where
Then we made a temporary plan
for the optimal LED wall
Through this, we planned the
optimal designs of driving comp,
environment shots, and LED walls that need large scale buildings
The final design was one that
goes about 180° in a circle
and a ceiling LED and reflection
or a moving LED wall that could
cover parts that need to be covered up
The most important part about this LED wall is
of course scene analysis
The most important thing is how must the LED wall be built
to be used as much as possible
But another thing one mus not forget
when working on a project is
the cost
As the number of LED wall increases
and the volume enlarges
the cost needed for that
also multiplies
so it took a lot of time to
find the optimal LED wall design
So there were some shots
that we had to give up on
You are seeing the built LED wall
As you can see, as we had planned,
there were not much challenges
until building the LED walls
The LED volume was built following the LED design
that we had made through storyboard and scenario analysis
From that point, we tested the LED volume and
prepared for the asset, that is, rehearsal
Another thing we thought about
before building the LED volume was
the connection with the set
The LED volume isn't built outside
but within the set
and therefore, depending on
how we build it in the set
the movement of the staffs, actors
and cars would differ
so taking these into consideration
we simulated whether to put it
diagonally or facing the front
In the end, we built an
LED wall facing the front
The second thing we decided on
was the image processor
to be used in the LED wall
Image processor is a device
used to play videos or 3D data
on LED walls
LED wall simply is a lighting device
so what color of light each device must shine
is ordered
by an image processor
An image processor takes HDMI or
SDI video signal as inputs
and analyses this video signal and
sends it off to each LED wall through a network
to tell what color to show
Depending on the LED processor,
the provided functions differ
Certain LED processor
only screens videos
What we wanted was that it should screen videos
but also needed the function of sync
and the colors expressed
must be equivalent to
that of an HDR filming
With all this into consideration, we chose our device,
and started preparing the devices
What was next was
camera tracking
It was also a very important matter
to plan how to do camera tracking
when it came to doing ICVFX
in front of 3D walls
What we would film
could be environments
and could also be driving shots
We might have needed fixed environments
or a wider shot
All this into consideration
and also the placement of the LED wall
optical camera tracking seemed
better for our circumstances
compared to gyro or tracking a marker
so we built optical camera tracking devices
and software system
We did also put up an
optical camera
for the LED wall as well
and we did several tests on various systems
to use lens data
We tried using lens data of
Ncam which was
one of the simulation cameras
and we also tested using Arri's track data
to simulate lens data
Then we also looked for ways to
establish sync which I stressed
that it is important earlier
Sync means that all devices must
draw the same frame at the same time
Not only the software that does the rendering
but also the camera that films must film the same frame
for there to be no scanning line on screen
or to not have empty areas on the screen
and have a perfect plate
But the problem is that the device that forms sync
is usually not wireless
There are a lot of movements in the camera system
and to have another wired device to be added in there
there are a lot of constraints for the filming staffs
and that in turns reduces the efficiency
so we prepared filming by
trying our best to
make the sync system wireless
and putting focus on
still connecting it to cameras
Ultimately, the rendering machine,
media wall, and camera tracking devices were
connected with wires
but the camera doing the actual filming
was made wireless and the sync system was built
Lastly, with all these
devices ready, the y have
to be connected to
software and devices
in the game engine
What you're seeing now is a screen called Ndisplay config
and what this shows is an environment
that can determined and set
how the LED wall media server
and image processor are used
to show what layout,
and what rendering machine to use
We also found the optimal setting
by testing various settings on the
LED wall, image processor,
and each rendering machine that we had built
If you have used a camera in front of an LED wall
you might know about the
characteristics of LED walls
LED wall often shows completely different colors
when seen by human eyes
and camera image sensors
The reason is that human eyes
and camera image sensors
have different ways of taking light in
so it happens inevitably
To solve this
the color must be edited based on how it shows
on the camera image sensor
What you see now is making a new color data
based on the image that shows
on the camera image sensor
and this shows how that color data is
set and applied on game engines
So the picture on the right
shows how the LED wall shows different colors on each sides
One side has the color data applied
and the other does not
So when preparing ICVFX
a lot of people fail to consider
this part about colors
but this part comes of different
for the type of camera or lens
so I recommend that you do
a lot of preparation and testing beforehand
to have a setting that can be applied
Next is matching
the virtual 3D space
and the real 3D space
What this means is
what becomes the standard of the virtual world is
the place and size of the LED wall
And the location from the LED wall
again becomes the standard to
setting the camera
Since it is impossible to
scan the whole set with a lidar
and put it in as 3D data to make the environment
we take the LED wall as the standard point
and from that LED wall, a 3D space
is calculated
You are seeing a
method of using Aruco Makrer and camera system
to transfer the camera location and LED wall
into a 3D space
Without a calibration process like this
the location of the real camera
and the real camera lens data might
look completely different in the virtual space
so there will be a lot of difficulties in the filming process
Next thing we tested is lens focusing, that is,
lens data
If the lens' focus, iris diaphragm, shutter speed
and such aren't simulated
the image rendered in the virtual space
will end up looking
awkward and disassociated
So simulation data must be made
of all lenses used in production
and each data must also have
a way to generate that in the 3D space
We tested varioius ways
and found the optimal one
and aimed to find a setting
that had the least burden on the camera,
especially for the camera director
Once the preparation is almost done,
a test is done with some key staffs
This isn't a prelight
but a test to see
what functions work and
to what extent it works
It is a rehearsal for the rehearsal
in a way
Once the asset achieves a decent quality
and the environment is ready
and the forementioned sync, camera tracking and such are ready
we gather with the key staffs
and check to what extent it can be filmed
and once again decided on
the quality of assets or filming direction
Once these functional tests are done
now we get into
the actual 3D environment developing
Artists keep updating the environment
and the updated data are
received at site through a
live version control system
and continuously played
in front of a camera and edited
to achieve the final quality
During that process, the data filmed
in front of the LED wall
are also worked on together with the filming team
and since we are creating a large scale environment
it is not done only within
but also single asset or environment
might be done with a co-working company
In these instances as well, version management
control system is used to send data back and forth and check them
The environment made is previewed
and continuously viewed in 3D environments
and checked
Then finally, through a process of prelight
we do a rehearsal
Prelight is a process of setting all the lightings
that will be used in the actual filming
and just using doubles for the actors
As you can see, for Black Knight,
for the first driving comp shot
we put all lights on the ceiling
and with all other lighting and camera team
we did a test shoot
In this rehearsal stage, we do a test shoot
for everything in the storyboard
We have extras instead of the actors
and make a list of what shot is possible
and what is not, and also what shot needs
what modification
And we take out the impossible shots
or adjust the shot's angle
or the lighting
and if time allows,
adjust asset
during the rehearsal
The video you're seeing is a video of the actual rehearsal
and what was important here was
as you can see on the right
you can see the reflection on the window
but because of the size of the LED wall
the reflection stops at the top
looking just black
Fixing this part
or removing this part
by changing the camera angle
These things are what's
determined during the rehearsal
Also in the rehearsal
various angles are tested
so side angles, frontal angles,
and what problems frontal angle has
and such were also tested
Then lastly, we filmed
The problem while filming Black Knight
was as you can see the filming using LED walls
was not done in a short period of time
but with long breaks
The reason why this becomes a problem is
LED walls have the advantage of cutting costs
and efficiency
but since the environment was ready but the scenes had to be
taken separately due to the schedule
prelight period was prolonged
and rehearsal periods also had to be broken up
which became a problem
It was also an advantage
that we could focus on each environments
individually which let us highten the quality
but it nevertheless was a deterrence to
ICVFX's efficiency
The video you're watching is a function of virtual production
It is a video where we tested how this space would look
using a simulcam
before the ICVFX
the reason why I'm showing you this
is because that space
also used the same game engine as the one used when making
the test video before prelight
so the simulcam could be used
in the site as well
Then the actual filming took place
As you can see, during the production
the score we give
are how many cuts were took
and how many cuts were
actually used in the final
The ones on the backgrounds,
the majority were shot
but indoors objects or lighting
needed a lot of post-editing
which makes the scores seem low
but we were able to do a
quite successful shoot
There were problems as well
The LED wall that we made for reflections
being not big enough
was checked only after the car was in the set
Even during the rehearsal, the big truck
couldn't be in the set
so it came in only the day right before the actual shooting
We couldn't calculate
the difference between the
truck's side wall size
and a normal car's side wall size
so the side wall did
have a problem of the reflection not being enough
And another thing was that the side wall,
which we thought would be easy to move around actually wasn't,
but it also broke
The second shoot
was a natural environment shoot
This one as well, some problems were
most of them were wide angle or tight shots
and we focused more on
wide angles rather than on tight shots
but the LED wall size
was squeezed into our schedule and budget
which made the LED wall not big enough
So we had to
edit it to look longer
after shooting
As you can see, comparing the prepared LED wall
and the actual filmed image's 3D scan,
the LED wall is smaller
so that is something that we could have improved
What was shot last
was air road
The biggest problem of air road was
driving comp
that took place in the company
or the CEO room
did not need floor extension
But air road needed a scene
where a person walked out of that huge place
and we needed extended floor art settings
But choosing the texture of the floor
itself was done pretty late to the schedule
and what we wanted was
at least a week or two
before the shooting
so that we can have that texture
on the 3D space
but that couldn't happen
So the actual LED space and connected floor
did not look the most natural
So it was a shoot which had some difficulties
in making a natural looking environment
To give you one tip
how we did the shoot in a rush
was that we used lidar scan to scan the floor
and brought the scanned texture as much as possible to unreal engine
and used the method of blending it
to connect it
Later on, doing ICVFX projects
if you do floor connecting
the biggest problem is having a source to
blend with the floor like that is possible
That isn't too hard
But the thing is, if you film with a camera, there will be a problem
What that is is
if we only film with camera A
using just one 25mm lens
you can just blend it from
that placement and that background
but if you use many different lenses
you have to change the blending for each lens
But that will take a long time
and doing that itself
needs all the light
so it can only be done during the prelight stage
So for floor connecting project
I recommend that you have a longer prelight stage
Lastly, these are the percentage information
on the total shot numbers of Black Knight
and the successful ones
Generally, it did shoot more shots
compared to The Silent Sea
but the schedule was stretch a few times in the middle
so I wouldn't say that this is an example of a 100% success
But what we felt was that in-camera VFX
is a technology that can
make a better quality content
if there is enough time given for preparation
and more communication
I told you about ICVFX
and through an example
I told you how it goes
Through this lecture
I hope you can get an idea of
how realistic content data is
used in actual movie production sites
Thank you
Virtual production project example
Production process Scene analysis, connection with the set, camera tracking
Building sync, connection with software and devices Color editing, matching virtual 3D space and real 3D space
Lens focusing, 3D environment production, rehearsal and shooting
Expressing Particles on colliding locations
A method of giving a Particle effect when Actor collides Search Destroy Actor after dragging Other Actor with mouse
Particle refers to the effect which happens when collision happens Content Browser, Content, Starter Content, Particles
Select Spawn Emitter at Location node and choose where to show Location effect
Be sure to separate what to proceed first when producing node