Hello, this is Juyoung Lim of Westworld
In this lecture, I will talk about
2D driving comp, an area of in-camera VFX,
and through examples, how they are shot
and used will also be explained
Workflow of 2D driving comp technology
I recommend that you learn about the most used
ICVFX technology which is 2D driving comp
about its general workflow
and how it's prepared and shot
through example cases
First, let me remind you what 2D driving comp is
It seems that driving scenes became a must in all drama series or movies
This is because there are a lot of instances where explanation, dialog, and
episodes take place while moving to somewhere
Before, to film these scenes
they brought cars into green studio
or put cars on tow trucks
to shoot these shots
If you film in a green studio
all works have to be done through compositing
and the quality or problem of spill
meaning green color showing off on other objects
might become an issue
Using tow trucks
have less flexibility in filming time and safety
But with 2D driving comp using LED wall
time and spatial limitation
and the advantages of driving comp
were enhanced in the environment
The biggest problem in driving comp is
the absence of a source to use driving comp
Driving comp needs a lot of source from
a certain road, environment, or area
Even a simple dialog needs to take the speed the car is going
and put that into consideration
which means that a lot of source is needed
and that it requires a
video shot in 360°
You might think that
a video shot with an action cam or
a VR camera would do for the 360° video
But images shot with VR cameras
might have the total resolution of 10K or 12K
but each video,
that is, the area filmed by the camera
might not even reach 2K
For this reason, when making a source for 2D driving comp
using an indoor camera is recommended
So the most important thing of 2D driving comp is
how to capture the source
and also what spec that captured source has
As you can see here, the captured source
must have a data more than 10 bit
and must be saved in the format of raw data
which has the least damage
And the capture speed
means that if we film at 24FPS
the captured source must also be shot at 24FPS
and if the movie is shot at 48FPS
the captured source must be captured at 48FPS as well
Only that way there will be no empty frames
and all frames will be perfectly captured
So it is important at what level of resolution
the provided camera films
Most action cameras or VR cameras
don't provide this high-end data format
So it is inevitable to have the challenge of
using many cameras and film at the same time
and rig them
I will give a more detailed explanation on
capture frame rate
Capture frame rate is what speed
a scene is captured in
If it's 24 frames in a second
it is taking 24 photos in 1 second
and making a 1 second long video
The reason why this is important is we take this captured source
and combine it with LED walls and film
If a source filmed at 24FPS
is converted to 48 FPS
there won't be empty frames
but there will be paused frames
A finger would have moved
but it wouldn't move in certain frames
This might not be notifiable to human eye
but it is inevitable that
a pausing video is film if a 48FPS camera
films simply a converting video
So the captured frame
must be identical or more
than the frame rate that is
used in the actual production
This is more about how much
the camera that is currently being used
provides rather than about development
The second thing is sync
Sync here means
if many cameras film at the same time
that they must film the same scene on the same frame
Same frame doesn't mean the same spot
but that when I capture a scene
another camera in another angle must capture the same scene in the same time axis
To achieve this
we need the sync system I mentioned in ICVFX
But except for these major cameras
there aren't much cameras that provide this sync system
But there is no other way than using cameras that provide sync system
and along with the sync system
we also need a system that can send off
a signal to start shooting
If the shoot is done with 8, 9, or 11 cameras
if the record buttons aren't pressed at the same time
the sync will be off from the beginning
so there needs to be a system that can check and control this
The third thing is
how to place the cameras
Camera placement, as you saw just before,
can be outside the car
or it could be in a circle
Which one is better depends on the project or filming method
The circular plate method that we use
is good for when we combine the images
filmed in many cameras into a single image
But since the cameras are connected to each other
there are less space, and controlling
the heat produced or the wind can be difficult
There will also be the issue of
how to attach this rig with 9 cameras
to the car or to a certain other object
Attaching cameras to the outside of a car one by one
might be convenient for filming and attaching
but for combining the filmed data into one
it will require more labor and effort
Because stitching must
be centered around and combined into one based on one origin
but each camera will have different origins
so there needs a process of adjusting the origins
What you're watching now is a video of when stitching is done,
what human eyes see, what a normal player shows,
and what a LED wall shows
Another problems is that when stitching is done
there is camera ghosting
Camera ghosting is a problem that occurs from the angle of camera array
Like the bus you see there
if two cameras filmed that bus and stitched them
there will be a slight difference in the angle
which would create some awkwardess
How to solve this problem
or how you're going to hide this
is another thing you must think about
Once it is determined how to film the image plate
scheduling for the project takes place
The image you're seeing is a schedule table
for a general 2D driving or ICVFX using image plate
Unlike 3D ICVFX,
filming step of asset preparation stage
and testing step of LED volume is added
From this, scheduling is done by calculating the time that will take to film
and test unlike 3D ICVFX
Examples of 2D driving comp
Based on the things I just explained
I will give a more detailed explanation
using D.P. season 2's driving comp
and train composition scene that we did
D.P. season 2 is a Netflix project released last year
and the shoot was during the end of 2022
And after the shooting and some post-production processes
it was released on July last year
You can consider D.P. season 2 project is a project
that goes through all steps of a driving comp
It was the first case where normal car,
several cars' interaction, and lastly a long train's driving comp shot
were all tested and actually filmed
to make the final plate
Through this, you will be able to learn
how sources must be captured,
how to prepare the captured source,
and what to do when the captured source isn't enough
D.P. season 2 took about 3 months of preparation
before the production
It wasn't really that short
and thankfully we had accumulated a decent amount of data
on car driving scene capturing
during another project we had done before D.P. season 2
And we also had the equipment needed
so the R&D of car scene was pretty much ready
but the R&D for the train scene which was the main thing
and solving technical issues spent a lot of the R&D cost
Next was filming and source production for the empty parts
So it was a project reaching towards the actual production
What's different from 3D ICVFX I explained on the third lecture is
is that 2D ICVFX only had a virtual production supervisor
This was because the team in charge of VFX was
a different team from us
so it was another challenge
to do virtual production working together with that team
So we had a more departmentalized structure
and we divided into teams that does
virtual production and a team that
processes sources and worked in a dual structure
What we did first was storyboard analysis
We actually only had
the plan that we were going to film car and train scenes
and didn't have anything planned for what scene to shoot for what episode
So we analyzed the storyboard
coming up with plans that
we could film this kind of shots but not these ones
and we scheduled to do
a tech-viz based on the storyboard
Of course there were some challenges
but we proposed that we do as much driving comp as possible
Once what scene would be shot
through storyboard analysis was done
we started talking about LED volume design
Volume for the car scene
was possible to cover
with the volume used in Black Knight
But the train scene
would need way more than the one used in Black Knight
So we used various tech-viz methods
to test what would be able to cover for the most area
How we could place the train
to use the Black Knight LED volume
If only one side of the train had the windows open
if that could minimize the settings
For car scenes, how the setting must be done depending on the size of the cars
We took all these into consideration
and searched for the final LED design volume
What was good here was
when we were doing this tech-viz
we used a media engine instead of game engines
Using the 3D data import function of the media engine,
as you can see, we didn't only bring the LED wall
but also the car that will be used
and the mesh of the train were also brought in
to calculate the size of the window
and the glass to be used
which is the function that helped us
greatly when designing the LED volume
We continuously tested various functions
and had options like maybe a straight setting will be better
or since the truck will be there
if might be alright to keep the circular structure
Especially how the source was screened was a very important issue
Car sources usually
screen a circular array
but trains are longer so we wondered if
it should be shot longer and wider or have a different array
These things we also had to think about
so we designed the LED volume thinking about
how the train would look on screen
if we put it parallelly
and how the time axis would look
So at the end, for the car scene,
it was decided to use the original LED
and for the train scene we decided to form a completely new LED volume
So in the train most actions would take place in the aisle
and the angle itself focuses on certain characters
so we covered one side as much as possible
and decided to block the other side where it reflects
and where there is no LED in the middle
we used curtains or people to
cover that part up
This final design
would have of course been better
if we could have LED on both sides
but to give the best effect within the given resource
this was what we settled on
And with this design
we did the pre-viz
What cource would play
and how they would be placed
How much it actually covers
and what windows need curtains over them
and where actors must be
These were all tested
Once this test was done
an actual train was placed
I had never known
that there were trucks to move trains
and that trains can be lifted up with craines
I also got to know that train wheels are also detachable
Anyways we started with
bringing a giant train into the studio
The reason why we brought the train in first before building the LED walls
was because we needed the train to get the right angle for the LED wall to be
and because there was no way for the train to be brought in once the LED wall would be built
So we started with bringing
the train into the set
After that we started building the LED
and it followed the design
that we had made
It took about a week to set the LED up
and after they were built, it took another week to
synchronize and test the LED
While the LED volume was being built
we did tech-viz in the meantime
The tech-viz combined and used
media server and
real time camera tracking technology used in real time preview
The yellow area in the video
are the parts excluding the actual chairs in the train that can move
and the green area are the space to the outer wall of the train
We tested with an actual camera
how much within this space could be covered
with the built LED and for the parts
that can't be, how we would need to
change the camera angle
This was the first case to use
media server and real time camera tracking
and through this, we could check that
not only game engines but also
media servers and 2D plates can make use of
real time camera tracking technology, that real time previews have
Next was testing how to produce sources using the train
We did various tests like if the 180° video just needs to be
filmed from windows on one side of the train or if it's better to film from different windows
with some time difference
What was used in this test was game engine technology
We made a virtual area identical to the train in the game engine
and arrayed cameras that simulate real physical cameras
and then checked how it would look
when they were composed at the end
with the videos filmed while the camera was moving
Even though there were plenty of windows in the train
we were able to confirm that we must stitch video
taken from one spot
but since this caused too much distortion
we thought of ways to decrease
this distortion
Based on these various data
we started gathering real production source data
There were cars prepared for
360° shots
but the majority of cars we would film
included sports cars, trucks, and SUVs
SUVs or trucks could use sources
filmed from normal cars with 360° filming
but sports cars, even if it was filmed from itself,
when we calculated the height of the camera rig,
the height that is captured in
didn't match
So for sports cars we had to split them into two 180° shots
and put cameras on the tow trucks
And as we were filming
we had to race on dirt road
instead of normal paved roads
And being on those unpaved surfaces
cameras would shake too much
and the shaking wasn't in the same lane
It even caused the camera array to be break down
so it would not match each other when doing the stitching
or more camera ghosting would happen
So to help this
we added an equipment called damper
so that it would absorb most of the shock
With this updated rig, the car plate was filmed
What you're seeing is the videos took with the updated rig and set
As we were on those dirt roads
without the rig that we had made that had dampers
the camera would keep falling apart
and stitching wouldn't be possible
but since we prepared pretty well
the shooting was done safely
So we took car source scenes from a total of 3 areas
They mostly had backgrounds of
demilitarized zones connected to North Korea
so we shot these shots in dirt roads,
bushy areas, and where there are mountains
I assume that D.P. season to would be
the first in Korea to film in these
rough places for driving comp
Usually it is done in Gangnam-daero or Gwanghwamun
where the night view is pretty
but since this one had the theme of the military
we captured a lot of sources from these unpaved roads
Like this, car sources were pretty much ready
and next
we had to gather sources for the train
There wasn't a way to attach cameras to the exterior of the train
Even Mugunghwa-ho,
we couldn't film from outside like we could before
We had to film it only from the inside
Then how to we attain source from the inside?
This was the challenge
We were going to film through the window
but how we would rig to the window
and how we would minimize the artifacts
were tested
Also, what was important in the train scene was the schedule
We would have to get on the train
For car scenes, for car sources
we just would have to tell the driver
Please go that way from the Gwanghwamun crossroad
Go 50km/h in Gangnam-daero please
Stop here then go there please
All these can be controlled
but for trains, that wasn't the case
If it departs from Seoul station, it could only
go to our destination in a constant speed
So then what was important was we had to have a detailed plan
on how to prepare at what timing
and film at what timing
As you can see we get on the train at Seoul station
and rig the camera and prepare
Wipe the windows
Get of where, all these were prepared
so that we could gather the sources we want following this schedule
And to make this schedule
we took trains from Seoul Station to Daejeon about 3 times
to check the actual time
About this much after we see country sites
About this much after we arrive at Cheonan-Asan Station
About this much after we're past Cheonan-Asan Station and see cities
and then lastly we enter Daejeon this way
We checked all these and made the schedule
Next was camera rigging
We tested for rigs to stick to windows
This again couldn't be tested on the actual shooting
so we took train from Seoul to Busan
and took it on and off from the windows
It stays on for about this long
It takes about this long
we checked these things
And the most important challenge was
once we did the filming and got the data
we had the issue that we had to keep the windows clean
We initially had thought that
if the sun was out, the outside would be captured with the camera
but since we had the focus as open as possible
all the dirt on the window were noticeable
So finding the spot with the cleanest window was the first issue
and the second was to keep it as clean as possible
So finding solution for the lighting on the window
and keeping the window clean
For that we even got a person to wipe the window
that would get off on each station
to wipe the window and then rig the camera back again
Continuing this process
we did the source capturing
After that, with the captured data
we finally did the stitching
From here, we used compositing software and matching of data
to see how cameras were arrayed
and how it looks in that array
How to fill in the empty areas
and minimizing camera ghosting
We found the optimal point and did the compositing
Here, finding the optimal value was a challenge
but the source we captured was about 25 minutes in length
They were all captured in 24FPS
so the frame amount was a lot
and rendering and combining frames
also took a lot of resource
And for D.P. season 2, what's interesting is
we also made environments using game engines
The reason was
one of the scenes the director wanted was
going quickly from a bright place to a dark place
He wanted a tunnel scene
But there wan't a long enough tunnel in Korea
Especially in the route from Seould to Daejeon
there wasn't any long tunnel
and it was pretty much impossible to adjust the iris diaphragm and film inside a tunnel
because it was too dark
So we made the tunnel section in 3D
and we proposed that we combine that section
with 2D captured images and play them together
For that, we made a virtual tunnel
and made the same rigging system that we had on the train
and attached it virtually to film that scene
Through this, we could produce a scene
where it seamlessly moved from a 2D source to a tunnel source
What you see on the left is an actual tunnel that we filmed
and one on the left is a tunnel video made in 3D
Real tunnels were too dark to show anyting at all
but the virtual tunnel had some lighting
which created the feel of the interior of a tunnel
and we could adjust the placement or brightness of the lighting as well
so we prepared the
tunnel sources in 3D
Another problem that occurred when preparing the source was
we usually filmed driving comps on roads
and when the weather is sunny
and at nights, when the sky was clear
But this time, there were a lot of unpaved roads
and they also wanted the weather to be gloomy as well
so it was very windy
That caused two problems
Since it was windy
the fan in the cameras would stop
and the camera would get heated
causing it to shut down
That made us stop the shooting
and turning it back on extended the source capturing time
So after shooting like this one day
we made a wind shield
so that no wind gets in the way
Of course it wouldn't be the best at keeping the camera cool
but keeping the fan going
was important for keeping the camera cool
so we made this wind shield and used it
And now we have this rig format
that we can attach wind shields on it
This as well, if we hadn't done this shoot
we wouldn't have known about this
but trying capturing sources from poor conditions
helped us discover this issue
Then we finally started the rehearsal
The most important thing during the rehearsal
was the interaction between the two cars
We had background footage we filmed with one car
but having two cars arrayed
was something we couldn't really guess how it would look
And would we be able to keep the speed the same while actually driving
Would we be able to drive them to look like one is racing
and the one behind is trying to overtake
These things were tested
Would it be better to push the car
or would it be better to put it on a motion base
Should we push the first car to the back
or should we push the second car to the front
All these things were thought about
and checking the actual backgrounds and sources
we looked for the right timing
Another thing the director wanted was a foggy area
One of the reasons why we shot on a dark day was
we thought that the day must be cloudy to have a foggy looking lighting
So we filmed a lot of the sources before and after it had rained
and in the site we sprayed fog
so that when we filmed it
with a camera, it would look like it was foggy
Now that the shoot is done
I think the fog helped a lot
because LED wall filming is no matter 3D or 2D
it is projecting something on a flat surface
So it looks more realistic when there are more layers in front of it
Having a lot of layer means having a lot of real objects
and when the camera captures objects on
the LED wall after real objects
the spatial perception from the layers
also create spatial perception for the background as well
D.P. season 2 had only two cars
but the fog created another layer
which created a more realistic spatial perception
We rehearsed the train scene as well
That scene had backgrounds
but the speed of the tunnel scene,
the speed of the lighting the height of it
These were the focus of the tests
and through the rehearsal we settled on where the lights should be
how bright the lights must be
and how far each lights must be
from each other
We brought this back to the unreal engine
and made the final source by re-rendering it
We also did a lot of testing on reflection
During Black Knight as well
while shooting Black Knight that we talked about in the third lecture
there were some issues on the reflection lacking
so this time we tried to find the angle that shows the most reflection
and searched for the right location for the car
Instead, we focused on finding lighting values
That could compromise reflection and moiré
Moiré is
the unique pattern of LED showing
LED is devices lighting up
so when those devices are captured by camera
it sometimes makes wavy patterns
and this is happens more often in reflections
So we tried to minimize moiré in the reflection
and searched for the widest reflection area
which was a challenge that we took during the D.P. season 2 shoot
One thing we earned from this was
LED stages are
being made and used in many occasions
but the LED angle, camera shutter opening angle
that we gained from this shoot can be used
as references no matter what the LED brand or spec is
so I think this was a good experience
Rehearsal for the train was also done once the train was in the set
The speed and color of the sources were the main things
but there was just one problem
You will see later but
right after we enter Daejeon station there was a shade
So we had set the camera for outdoors
and the moment the shade was there
the screen would turn extremely bright or dark
because the iris diaphragm wasn't right
We didn't have auto iris diaphragm
so it was inevitable
but the director wanted the lighting
to be the same even when the shade appeared
after the rehearsal
But we didn't have the time to film it again
so we had to color edit that part
to brighten it a little bit
and make it less awkward when entering Daejeon station
Looking at the final product it isn't too noticeable
but the colors do seem
a little bit burnt off
because there was too less light
That was another issue that we discovered during the rehearsal
I assumed that if we have another situation
where we enter an environment that suddenly gets dark
we would use two sets of cameras
We would set one set to the outdoors
and the other to the darker inside
to get both sources
will be what we'll do
Then we get into the production
This project was unique because
the team doing the post-production
and the virtual production were different teams
and the most important thing was
the communication between the targeted direction of the post-production
and how we should do our part
So we established a way to
periodically communicate with the VFX supervisor
and through this I could see that working together with
another company is possible in a project
You're seeing the train scene shooting
Many different angles
The advantage of the train scene was
that we could film both directions of the actors
and the tunnel effect was also
implemented without an issue
Unlike cars
there were no close ups on the windows
so other artifacts shown on the LED
like light smudging or some moiré forming
wasn't noticeable at all
I think we could focus on the scene itself for the train scenes
For car scenes, the problem of how to move
the two cars that I mentioned earlier
also didn't have much issues since the sprayed fog
created another layer and made the shot
look much more natural
What you're seeing is the second two car scene
where you can see the sports car pass by
That scene looks interesting even now
because the speed of the sports car and the background don't match at all
That scene we didn't settle on what sources to use in the rehearsal
and chose only at the last moment
so the source's frame rate and the
speed wasn't determined at all
which made it a bit awkward
I think that it was a case that
let us know how to set the sources
for cars going in completely
different speeds
This was a summary of D.P. season 2
which is a successful ICVFX case
All shots were given OKs
and all shots were used in the final without editing afterwards
I think this was a project that proved that
for projects like D.P. season 2 with
large scale trains or driving scenes
not only simple dialog but also action or car chasing scenes
can be done in 2D driving comp
I think it is a technology
that can be used in many other contents with the lessons we took from this
I explained about 2D driving comp in this lecture
2D driving comp is in a way
combining ICVFX and sources from existing filming methods
which shows a new production method
while also being the most applicable method
Like this
I hope you can combine the technology you are learning now,
existing filming methods and the content being created
to produce the highest quality
in the shortest time
Thank you for listening
Workflow of 2D driving comp technology Problems of driving comp Absence of source to use driving comp Sync Camera placement
2D driving comp examples Production steps Storyboard analysis LED volume Tech-viz and testing Source production and preparation using trains Rehearsal Production