Realistic
Realistic Advanced Concept for ICVFX System Configuration
GCC Academy
Hello everyone, I am Park Ji-yong, in charge of ICVFX Integrated App Flow
The lecture on the integrated ICVFX workflow is divided into four sessions
It will cover the overall technology for implementing in-camera prepass using a real-time game engine
This includes fundamental concepts for configuring an ICVFX system,
collaboration and version control for practical use,
multi-display setup using LED displays,
and real-time collaboration and control settings
I encourage you to build on the concepts learned in this lecture
Understanding VP
Before discussing virtual production, let’s briefly review traditional filming methods
The first method is on-location shooting,
where filming takes place at real-world locations,
ranging from urban settings to remote natural environment
This approach provides diverse background options
For instance, films and dramas set in modern times
are often shot in city locations
While this method allows for realistic imagery,
it is highly affected by external factors, such as weather and time of day,
making it difficult to control the shooting environment
The second method is open set shooting,
where backgrounds and props are physically built for filming
This approach is commonly used in films, dramas, period pieces, and fantasy scenes
Lastly, there is chroma key shooting,
which involves filming in front of a green screen
and replacing the background in post-production
This technique offers great flexibility in creating
various locations and settings in post-processing
However, it heavily depends on post-production,
making the workflow complex and potentially increasing costs
Now, what exactly is virtual production?
Many might think it refers only to ICVFX using LED walls,
but virtual production is an umbrella term covering various new aspects of media and entertainment
VP can refer to specific terms or a combination of them
Let’s briefly go over each one
First is the previously mentioned ICVFX
This method projects a virtual background
directly onto the camera through an LED wall
Second is motion capture, or mocap
Mocap captures the real-time movements of actors or objects
and applies them to virtual characters or objects
Third is previsualization, or previs
This visualizes scenes in 3D simulation before shooting,
allowing pre-planning of composition, movement, and lighting
By simulating in advance,
it helps detect and resolve unexpected issues before filming,
making it highly efficient
Next is VR
VR is useful for exploring virtual sets
or simulating shooting plans in advance
A director can wear a VR headset to navigate the set and adjust plans beforehand,
a process called virtual scouting
Next is MR
MR provides a mixed environment of reality and virtual elements,
allowing actors to interact with virtual objects in real time
Its key feature is the seamless integration of virtual and real elements
Additionally, there is XR
XR integrates AR, VR, and MR to provide an extended experience
that bridges virtual and real environments
This enhances immersion
Next is the virtual camera function
A virtual camera allows free exploration of angles and movements in a virtual environment
By integrating mocap data,
it helps pre-determine optimal angles
before actual filming
Lastly is Simulcam
As an extended concept of AR,
Simulcam composites virtual elements into real-time filming scenes
By applying mocap data in real time,
it allows immediate visualization of how actors interact with virtual characters
Many of these technologies existed before the concept of VP emerged
Why have these technologies been grouped under Virtual Production?
Let’s explore the common characteristics of VP features
First, real-time interaction and feedback
VP connects virtual environments and cameras in real time, enabling immediate feedback
Second, the fusion of virtual and real environments
VP naturally integrates physical and virtual sets,
creating a seamless environment
This allows actors to interact naturally with virtual backgrounds
as if performing on a real set
Third, previsualization and optimized planning
VP allows 3D previews of scenes in advance,
helping adjust optimal angles and movements
This helps identify and fix potential issues before shooting
Thanks to these features, VP is increasingly becoming a viable option
that integrates well with traditional methods despite being a new technology
Technologies with these common characteristics are broadly included in VP
Now, let’s briefly look at three Virtual Production technologies using LED walls
First is In-Camera VFX, or ICVFX
This method is mainly used in films and dramas,
projecting real-time 3D backgrounds onto an LED wall for shooting
This allows actors to perform while seeing a realistic virtual background,
achieving high-quality scenes directly on set
Second is 2D In-Camera VFX
This method is also mainly used in film and drama production
It projects a pre-rendered 2D background video onto an LED wall
It is best suited for scenes that require minimal background movement
Since source production is relatively simple, it saves time and cost
while maintaining high image quality
Lastly, there is XR
XR is used in live events and broadcasting,
integrating LED walls with AR and VR technology
It creates an immersive environment,
particularly useful in interactive settings with audiences
These three technologies are chosen based on their purpose, film, drama, or live entertainment
Now, let’s explore each technology in more detail
In-Camera VFX relies on Unreal Engine, which is its key feature
Using Unreal Engine, high-quality 3D environments
can be rendered in real time and projected onto an LED wall
This allows for dynamic background changes,
immediate lighting and angle adjustments,
and real-time feedback on set
As seen in the video,
as the camera moves, the inner frustum moves accordingly,
dynamically adjusting the virtual background in real time
Now, let’s discuss 2D In-Camera VFX
2D In-Camera VFX involves projecting pre-recorded or pre-rendered 2D footage onto an LED wall
This method is mainly used for fixed backgrounds or specific scenery, such as these
A landscape seen through a window
The outside view during a car-driving scene
It enables easy background rendering
without complex 3D environments
Since setup and rendering are simpler,
it reduces time and cost
Additionally, as it uses real footage or pre-rendered videos,
it offers a high level of realism
Before diving into more details, let’s take a quick look at an ICVFX system diagram
The ICVFX system consists of a camera, tracking system, render machine, LED panels,
LED image processor, and sync generator
These components work together
to integrate the virtual environment into real camera footage
Each piece of equipment is connected by different cables,
transmitting large amounts of data and signals in sync
Understanding how these components interact is crucial for system setup and adjustments
Since ICVFX is a complex process,
any misconfiguration can cause system-wide errors
If this diagram seems unfamiliar now, don’t worry
After completing this course, you’ll find it easier to understand
So no need to worry about it too much
Now, let’s break it down further
This diagram represents the ICVFX system setup shown in the previous video
Each component is connected via various cables,
exchanging data and signals in real time
We will go through each component in detail,
but first, here’s a brief overview
The tracking system continuously tracks the camera’s movement,
lens focus, and zoom data, updating the virtual environment in real time
The OptiTrack cameras positioned to the left of the main camera are connected to a switch
These cameras detect the sensors attached to the main camera,
allowing for precise tracking of its position
The diagram shows a total of four render machines,
which play a crucial role in rendering real-time 3D graphics
and displaying them on the LED screen
Each render machine runs Unreal Engine, ensuring seamless real-time rendering of virtual environments
The LED screens and LED processors are responsible for outputting the final visuals
The render machines generate images in real-time,
which are then processed and displayed on large LED screens
The sync generator ensures all devices remain perfectly synchronized
by generating precise timing signals
Since numerous devices must operate simultaneously without any delay or misalignment,
synchronization is critical in ICVFX
Setup in ICVFX
To set up ICVFX, various hardware components are required
I will first introduce these key components briefly from the previously shown diagram
before explaining each in detail
The first major component is the LED wall,
which serves as the primary display for virtual backgrounds
It is installed on the actual filming set, providing both the background for actors
and the environment captured by the camera
Next is the image processor,
which converts the rendered data into a signal optimized for the LED wall
This device adjusts resolution, color, and brightness,
ensuring high-quality visuals on the display
The camera and lens system is also a critical component
In ICVFX production, the camera and lens must
be precisely tracked to align with the virtual background,
ensuring that real and digital elements blend seamlessly
The media server or render PC is another essential element
It runs software such as Disguise or Unreal Engine,
rendering 3D virtual environments in real-time
The genlock device manages synchronization between all components
In an ICVFX system, it is crucial that cameras, LED walls,
and render machines stay in sync, preventing visual discrepancies
The last key component is the camera tracking system and its related elements
This includes lens data streaming and six-axis tracking,
which updates camera position, lens focus, and zoom data in real time
on the virtual background
Each component of the ICVFX system functions in synchronization
Now, let’s take a closer look at each of these components
First, we will examine the various types of LED walls used in virtual production
LED walls generally come in three main configurations: flat, curved, and MR set
The choice depends on the specific purpose
Flat LED walls are the most basic type
They are relatively easy to install
and are best suited for shooting from specific angles
This setup is often used for interviews or static scenes with minimal camera movement
Recently, it has become popular for window background solutions in production
Curved LED walls are the most widely used configuration today
They provide a consistent background across various camera angles
and allow for smooth panning and rotation
They are particularly useful for driving scenes, as they can cover a wide reflection area
MR set LED walls include an LED floor,
extending the virtual background beyond the physical set
Even with a smaller LED setup, this approach allows for background extensions beyond the LED boundaries
Now, let’s briefly compare MR set LED walls and ICVFX LED walls
MR sets include LED floors,
as I mentioned earlier,
allowing for a fully extended virtual background that integrates the floor
ICVFX setups do not use LED floors Instead, they rely only on LED walls and ceilings
This distinction depends on the specific production requirement
In an MR set, the floor is also a digital element,
as a fully extended virtual background
while in ICVFX, practical props and set decorations
are used on the floor to enhance realism
Therefore, it is important to use a set without floor LEDs in ICVFX
Now, let’s take a look at the second component of the ICVFX system: the LED processor
The image processor is responsible for accurately transmitting the virtual background to the LED wall
It receives the rendered image from the render machine, distributes it properly to the LED screen,
and adjusts the color and brightness to manage the final image quality
Since image quality on the LED wall is crucial in ICVFX,
the image processor plays a key role
The most commonly used models include the Helios from Megapixel and the XS40 from Brompton
Each manufacturer offers image processors with different specifications,
allowing studios to choose based on their specific requirements and environment
The LED wall consists of multiple panels
For example, if an LED wall is built to be 20 meters wide and 6 meters high,
it would typically use 480 panels, each measuring 50 cm by 50 cm
Each panel functions as an individual display,
and the image processor ensures that all 480 panels
operate as a single large screen by distributing and linking all the data
Let’s take a quick look at the operating mechanism of an LED image processor through a diagram
As shown in the diagram, the image processor
transmits data to multiple small LED panels via network cables
It ensures that each panel accurately displays its portion of the overall image
Each panel functions like an independent monitor,
and the processor integrates them into a single cohesive display across the entire LED wall
This diagram illustrates how the image processor distributes data to each panel,
integrating the entire LED wall into a unified display
The next essential component for ICVFX shooting is the camera and lens
Not all cameras can be used,
and it is recommended to use models that provide specific features
Most cinema cameras are compatible,
but there are some key functions to consider
The first and most important feature is Genlock
Genlock ensures that the camera, LED wall, and all other equipment
are precisely synchronized for stable shooting
In ICVFX, multiple devices must operate simultaneously,
and if they are not perfectly synchronized,
the footage may become unstable
The second essential feature is a global shutter
or a high-speed rolling shutter
This is crucial for accurately capturing the virtual background displayed on the LED wall
Now, let’s move on to the render machine,
also known as the render PC, used in the ICVFX system
A render machine is not a specific piece of equipment
but rather a high-spec computer equipped with a powerful GPU
Since ICVFX requires real-time image output to the LED wall,
fast and stable rendering performance is essential
Typically, NVIDIA A6000
or A6000 ADA graphics cards are recommended
However, when dealing with extremely high-resolution LED walls,
a single render machine may struggle to handle the entire resolution
For example, let’s assume that the green box in the image represents an LED wall
with a resolution of 9504x1769 pixels,
approximately 10K resolution
Such a high resolution cannot be managed by a single computer,
so multiple render machines are used to divide the rendering workload
In the image, the red lines illustrate an example of dividing the LED wall into four sections
Each render machine is responsible for rendering an area of 2376x1769 pixels,
collectively achieving the full resolution
This method reduces the load on each render machine,
ensuring a stable video output
This is made possible by nDisplay, a multi-display system
provided by Unreal Engine
It allows multiple render machines to work together to form a single LED wall
In the upcoming practical session, we will directly observe the nDisplay function in action
While the render machine handles only the rendering process,
a separate operating computer is required to manage image control and content distribution
The operating computer is responsible for synchronizing and controlling all render machines,
ensuring that the entire system operates smoothly
To achieve high-resolution LED walls, multiple render machines are necessary
Using systems like nDisplay, the rendering workload can be distributed efficiently
Now, let’s take a look at the Genlock system, another key component of ICVFX
The diagram below provides an overview of Genlock synchronization within the system
This illustrates how the Genlock master device connects with cameras, render machines, and image processors,
ensuring system-wide synchronization
At the center of the system, the Genlock master device generates timing signals,
ensuring that all equipment operates in perfect synchronization
With this setup, the LED wall, camera, and render machines remain precisely aligned,
maintaining a seamless display
This perfect synchronization of all machines
prevents flickering and instability,
allowing for smooth and reliable ICVFX shooting
Next is NVIDIA Quadro Sync function, which enables Genlock synchronization across multiple render machines
As shown in the diagram, each render machine is connected via NVIDIA Quadro Sync,
allowing them to share a common Genlock signal and synchronize their rendering tasks
This function is crucial when multiple render machines are used together,
as it ensures that all machines operate in perfect sync
The following image illustrates how NVIDIA Quadro Sync is configured
This sync card connects to the graphics card
and links multiple computers using LAN cables,
forming a daisy-chain connection
The tracking system is one of the most
critical components of ICVFX
Its primary function is to track the camera’s position and lens data
and transmit this information in real-time to the virtual camera in Unreal Engine
This allows the virtual environment to match the exact viewpoint and framing of the real-world camera,
ensuring seamless integration between physical and virtual elements
The six-degree-of-freedom tracking method is used,
capturing movement in all four directions
and rotation to accurately reflect changes in the virtual space
With this, we have covered the key components of ICVFX systems
Next, we will explore each component in greater detail,
starting with shutter types and scanning methods
First, let’s look at an image comparing rolling shutter and global shutter
A rolling shutter captures an image line by line
from top to bottom in sequence
When capturing fast-moving objects, this can lead to skewed or wobbly effects often seen in high-speed footage
On the other hand, a global shutter exposes all pixels simultaneously,
preventing distortion and ensuring clear, accurate images even in fast-moving scenes
Next is genlocking
Each rendering machine in the ICVFX setup must be genlocked to stay synchronized
For example, let’s assume that two computers are responsible for rendering half of an LED wall each
Both computers must render their respective sections simultaneously and send the output to the LED panels
If the two computers are not genlocked,
the images will be out of sync, resulting in visible misalignment as seen in the right
When properly genlocked, both machines output frames at the exact same timing,
ensuring a synchronized image across the entire LED wall
This video shows an error that occurred during my genlock test
As you can see, we played a test video where the same color should appear sequentially
However, the colors do not display at the same time,
indicating that the sync is significantly off
How does genlock work?
Many assume that using timecode-based synchronization ensures everything stays in sync
However, without proper genlock synchronization, the system will not function correctly
Both LED panels and cameras operate using digital signals
For example, let’s say the LED panel outputs 24 frames per second,
and the camera captures at 24 frames per second
If these two signals are not genlocked,
their timing will gradually drift, as seen in this image
For a correct image, the LED must output frames at the exact same timing
that the camera captures them
You’ve likely seen genlock issues
in everyday situations
For example, when filming a TV screen with a smartphone,
you might notice rolling black bars or flickering
This flicker occurs because they are not synchronized,
or when there is no proper genlock
Without proper genlock, flickering or tearing will occur in the final image
To capture accurate and realistic images from an LED wall,
genlock is a critical component
Let’s take a closer look at how perspective changes depending on the camera’s distance from an object
The diagram illustrates how an image is formed based on the distance
between the camera center and an object
As the camera moves closer to the object,
the object appears larger when projected onto the image plane
This distance-to-projection ratio is the fundamental principle behind perspective formation
Here is a more detailed example
In the upper image, both the person in the midground and the tree in the background are visible together
However, as the camera moves closer, the lower image shows that the tree appears relatively smaller
The objects in the background shrink as the camera gets closer to the foreground
This is known as perspective distortion
This perspective effect is crucial in ICVFX
because it ensures that virtual backgrounds and physical props appear seamlessly aligned
Since the size and depth of captured images change
based on the camera’s distance and angle,
this principle helps maintain visual continuity between virtual and real elements
Thus, perspective is an essential concept for understanding
how camera position and distance affect an image
Now, let’s explore how perspective is applied in ICVFX
In ICVFX, perspective is handled using two zones on the LED wall: Inner Frustum and Outer Frustum
The left image shows the Unreal Engine interface,
where the green area represents the Inner Frustum
This zone is directly captured by the camera
and dynamically adjusts its position and angle based on the camera’s movement
Outside the green area, the remaining LED wall belongs to the Outer Frustum
This area is not directly captured by the camera,
but it plays a crucial role in providing accurate lighting and reflections on actors and objects in the scene
Without this system, filming in front of an LED screen would not be different from traditional green screen shooting,
resulting in an unnatural and less immersive final image
By combining real-time rendering and camera tracking,
ICVFX allows virtual backgrounds and physical environments to blend seamlessly,
creating a highly immersive experience
Now, let’s explore why lens data streaming
is crucial in ICVFX and how it works
So far, we’ve focused on camera position tracking,
but lens data also plays a vital role
If we consider a real-world scenario,
while the dog in the foreground and the tree in the background remain blurry,
only the child at 10m is in focus
Now, let’s apply this scenario to an LED wall
If the red line represents the LED screen,
the virtual background projected onto it will always remain in focus
because it exists on a flat plane at a single distance
This creates a problem because,
in real-world optics, the tree at 20m should be out of focus,
but on an LED wall, it appears unnaturally sharp
This discrepancy reduces realism in the final shot
To address this issue, ICVFX uses real-time lens data tracking
By calculating the distance between the camera and the virtual tree,
the system can intentionally blur the tree on the LED wall,
As a result, objects that would naturally be out of focus in reality
also appear out of focus in the virtual background,
ensuring consistent depth perception
By tracking camera position and lens data in real-time
and adjusting the focus and depth of images projected onto the LED wall,
ICVFX ensures highly realistic visuals
This technology allows virtual backgrounds and physical props
to blend naturally, resulting in high-quality scenes
Now, let’s discuss LED meshes, which are essential
for using NDisplay in ICVFX
To accurately project images onto different sections of an LED wall,
we need an exact shape and size of the physical LED wall
The LED mesh serves as the fundamental structure of a 3D model
and must be created to match the exact shape of the physical LED wall
Each mesh determines which render machine is responsible for which section of the LED wall
By importing this mesh into Unreal Engine,
precise area segmentation and rendering become possible
Therefore, accurately creating an LED mesh file
and properly configuring its UV mapping are crucial
for maintaining consistency between the virtual background and the real set,
thus enhancing immersion
Next, let’s look at camera tracking systems
In ICVFX, it is essential to synchronize physical camera movements
with the virtual background
There are two main tracking methods:
outside-in tracking and inside-out tracking
Outside-in tracking uses external cameras or sensors
to track objects inside a defined space
This setup allows for highly precise position tracking since the external device remains fixed,
and it has the advantage of covering a wide area
However, installation complexity and obstructions can affect accuracy,
making it ideal for fixed studio environments
Inside-out tracking, on the other hand, uses sensors or cameras attached to the camera rig
to track its position relative to the surroundings
This method is more portable and adaptable
to changing environments,
but its accuracy may be lower than outside-in tracking,
especially in complex settings
Finally, a markerless inside-out tracking system, such as NCam, can be used
NCam employs sensors that analyze real-time environmental data
and generate point clouds to track positions
The key advantage is that no markers are needed, and installation is quick,
though lighting variations may cause errors
While professional tracking systems offer
high precision and stability,
they are also very expensive,
making them difficult to acquire for individual users
However, if an individual wants to test ICVFX or use it for simple applications,
there are a few alternatives
For example, VR device tracking features, such as those found in Oculus Quest controllers
or Vive trackers, can be utilized
These devices provide lower accuracy
but serve as viable solutions for small projects
These devices may have lower precision compared to professional tracking equipment,
but they can still serve as practical alternatives for personal testing or small-scale projects
Now, let’s take a look at how the lens data streaming system is applied to Unreal Engine
The core of this system is the ARRI lens data tracking system,
which measures camera lens data, such as focus, zoom, and aperture, in real time
and transmits it to Unreal Engine
To achieve this, a device like UMC4 is required
This equipment collects lens data from the camera and transmits it to a computer
Then, using the Live Link function and the ARRI metadata plugin,
the data is streamed into Unreal Engine
By reflecting this data in real time,
the virtual environment and camera movements align perfectly,
enabling more natural visual effects
Now, let’s take a closer look at ARRI’s UMC4 and LDS
The UMC4 is a device designed to accurately stream lens data within the ARRI camera system
This device collects lens metadata and directly transmits it to Unreal Engine,
with data transmission occurring via a network cable
Understanding ICVFX Through a Diagram
So far, we have examined the key components and concepts necessary for ICVFX
Now, let’s go back to the initial ICVFX system diagram
and summarize how the entire system is structured and operates
Looking at the diagram, the core components of the ICVFX system are as follows
First, the operating machine
It controls and manages the entire system
It is responsible for network communication
and is connected to multiple rendering machines and network devices
Second, the rendering machines
It process real-time 3D environments using Unreal Engine
and output them to the LED wall
Third, a network switch hub and a sync generator synchronize all devices
These components ensure that signals between devices remain in sync,
preventing frame drops or synchronization errors
Fourth, the LED image processor
It optimizes the rendered images for display on the LED wall,
ensuring high-quality visual output
Finally, the camera tracking system and lens data tracking system
They ensure that the movements of the real camera and lens data are accurately reflected in the virtual environment
This synchronization helps create seamless and realistic scenes
With this, we now have a comprehensive understanding of the essential elements that make up ICVFX
This diagram illustrates how these components interact to integrate virtual environments
smoothly into real-world production
Next, we will apply everything we have learned
and explore how these components and equipment
are set up and connected in an actual studio
VX Studio On-Site Demonstration
We are now at VX Studio, located inside the Gwangju Immersive Content Cube or GCC
Here, we will demonstrate each piece of equipment discussed in the theoretical lecture
and provide a brief explanation of their functions
First, you can see a large-scale LED wall
This LED wall is controlled from the Brain Bar,
which serves as the central command center for ICVFX operations
Supervisors and operators manage and fine-tune the visuals displayed on the LED wall,
ensuring that they align with the director’s vision
Now, let’s take a closer look at each component in detail
The LED wall, which is the most critical element in in-camera VFX,
may appear as a single, large display
However, in reality, it is composed of multiple small modular display panels
that are combined to create a massive screen
Now, let’s take a quick look at this panel
The LED modules currently in use are typically
50cm × 50cm in size, and when combined, they form a single cabinet
As you can see, the distance between individual LED pixels is referred to as pixel pitch,
and the installed LED wall here has a pixel pitch of 2.6mm
At the back, there is a power board with dedicated connectors that allow multiple modules to be attached
Each power board connects to four modules,
forming a single panel,
which can then be expanded into a large LED wall
The LED wall you see here is designed for vertical installation,
but LED panels for ceilings and floors are a different type
Now, I have moved behind the LED wall
Let's take a closer look
From the front, it may not be obvious,
but each rectangular section here
represents a single LED module
Each cabinet is made up
of four LED modules
The more cabinets added, the larger the total LED display becomes
Looking at the cables, there are four main connections
Data in,
power in,
data out,
and power out
The biggest advantage of this system is that power and data are not supplied individually to each panel from a central unit
Instead, they are daisy-chained from a single input point
Because of this setup, if an issue occurs on one side of the system,
it may cause the entire chain to shut down
This modular structure is a key aspect of the system’s design
Additionally, if a single panel malfunctions,
it can be removed and replaced immediately
This makes maintenance much more efficient and convenient
Now, let’s move to the main control area and take a closer look
I have moved further back from the LED wall,
and as you can see, there are a huge number of racks here
This area serves as the central hub that powers and controls the entire LED wall
If you look here, you will see a small unit,
which is the power supply box responsible for delivering power to the entire LED wall
Next, these units here are Brompton’s XD processors
These devices distribute the image data received from the LED image processor
to the individual LED panels, ensuring that each receives the correct visual information
All of these connections are established through network cables
Now, let’s take a brief look at the back of these units
I have moved to the rear of the XD processors,
and as you can see, there are a large number of LAN cables here
To explain their roles briefly:
The cables neatly lined up at the top
transmit data to the LED panels
The fiber optic cables at the bottom of each device
receive image data from the LED image processor
This entire setup operates through a network infrastructure
Now, let’s move on to the LED image processor
As you can see, everything is structured like this from here onward
These devices are LED image processors, whose role is to calculate,
manage, and deliver images to the LED wall
In simple terms, they optimize video signals before sending them to the LED panels,
ensuring that the panels display images accurately
Looking at the front of the processor,
there are actually not many physical controls
In fact, there are only two main buttons,
but most operations can be managed remotely from a computer
Once these devices are installed in the server room, there is usually no need for frequent manual adjustments
Now, let’s move to the back of the image processors
and examine how they are configured
I have now moved to the rear of the image processor
This space is narrow and densely packed with cables,
but I’ll briefly explain the key components
The blue fiber optic cables here transmit data
from the image processor to the XD processors that we previously discussed
Each image processor sends data to four XD processors
Additionally, the green cables here carry the genlock signal
There are also other network cables here
that allow remote control of the entire system
Although the wiring might look complex,
everything is neatly arranged
The cables are carefully labeled and systematically organized
If any issue occurs, this structured setup
allows for quick identification
of the problematic cable, making repairs
and adjustments far more efficient
Now that we’ve covered the hardware aspects,
let’s move on to how these devices are controlled
and utilized on a computer
This time, I’ll demonstrate how the image processor
can be controlled via a computer
and explain its functions
and roles in detail
On the desktop, you can see the Tessera Remote software
This software is network-based, allowing remote control of the image processor
By clicking Start,
we can remotely access all image processors we saw in the server room
Here, the system is categorized into ceiling, main wall, and floor
Each is labeled, ceiling as ceiling
and floor as floor
For now, we’ll focus on controlling just the main wall
Once launched,
the UI appears, showing the main interface
I’ll briefly explain this UI
and then go over the specific features it offers
The UI is divided into tabs such as Project, Edit, View, and Tools
The Project tab
allows different users to manage
and configure image processor settings
based on their needs
The Edit tab enables modifications,
undo/redo actions,
and other configuration adjustments
Changes made here are not permanent until saved,
meaning we can revert to a previous state using Ctrl+Z if needed
So even if there was a mistake,
we can go back
I don't know if you can see this here,
but if I move it like this,
you can see a change on the LED screen,
which can be undone with Ctrl Z
This software essentially allows for LED wall configuration
These arrows indicate
how data is transmitted from one panel to the next,
in the form of data cable chains as we saw before
ensuring the entire LED wall functions as a single display
Below, we have
preset options for different projects
Since color settings and configurations vary across projects,
presets are used to store and apply settings efficiently
Now, let’s take a look at the most important section at the top
This area is quite intuitive
Input, processing,
how it's being done,
whether to override this,
and then finally the output
All these elements are accessible in one interface
In terms of input sources, as mentioned earlier,
the primary input used here is HDMI,
but SDI can also be connected
If we select the HDMI input,
click on it,
we can view key details such as
resolution, frame rate,
color settings, and bit depths
Based on the image color values being transmitted,
we can choose to use PQ (Perceptual Quantizer) or SDI settings
Additionally, we can select between Rec. 2020 or Rec. 709
Currently, this studio is using a custom color space
Once we verify the input settings,
we move to the processing stage, where additional adjustments can be made
For example, we can modify the resolution
and adjust basic color settings
However, in most cases, rather than directly from the image processor,
we recommend controlling these settings via the media server
In some cases, LUTs (Lookup Tables) can be applied for additional processing
Depending on the situation,
if we want to achieve more precise color calibration,
we can load a cube file following a specific color pipeline
Next, we have the Override and Freeze functions
In the Override section,
you can select a pattern to determine
which screen will be displayed during standby mode
Usually, we use a scrolling mode,
where a basic color bar continuously cycles on the display
Alternatively, the Freeze function is used
when we need to display a specific image on the LED screen while working on other tasks
Since the display would normally update in real time,
Freeze locks the screen to prevent changes
The Blackout function, on the other hand, completely turns off the display, making it go dark
The Nits adjustment allows us to control the brightness of the LED display,
increasing or decreasing it as needed
Here, you can see the network section
Each XD device we previously looked at
can be monitored individually, displaying its data processing status
However, if there is an issue, this green
they may turn red, signaling a problem
So that's the gist of it
Additionally, we can check the Genlock synchronization status
In this setup, we are using 59.94 Hz for Genlock
The system receives an external synchronization signal
via the Reference In input
This concludes the overview of the image processor
Throughout a project, this system remains active at all times
to monitor potential issues,
manage the display, and perform maintenance as needed
Now, let’s look at one of the most critical components of ICVFX,
which is camera tracking
At VX Studio, two primary tracking systems are in use
The first one
is Spider, which we have over here
Spider does not operate independently
If you look above the LED wall, you’ll see small cameras installed
These are called followers
These cameras detect and capture the light emitted from Spider
The biggest advantage of this system is that any camera
equipped with Spider can be tracked instantly
If you look closely, Spider has a circular design,
allowing cameras to seamlessly track motion
when placed within the system
Since Spider emits light,
it requires a power source,
and it must communicate with the lens system to relay positional data accurately
This concludes the basic explanation of Spider tracking
Now, let’s move on to the second tracking system
If we move over here,
you’ll see a Red Spy camera mounted on the rig
This camera system features a lens
and an LED emitter
Red Spy works by emitting light onto, though it's not visible,
reflective stickers placed on the ceiling
These stickers reflect the light back to the camera,
allowing it to calculate positional data
based on the reflection
A key advantage of this system is that, as long as the sticker placements remain unchanged,
the tracking system automatically functions whenever it’s turned on
Now, let’s move to the computer interface
to see how this data is processed
Let's go
Now, if you look at the screen here, this is the Red Spy mapping interface we just saw earlier
Each of these dots represents a sticker affixed to the ceiling
The system continuously calculates the positions of these stickers,
creating a comprehensive map for tracking
Let’s move the camera to demonstrate
As the camera moves,
these dots are staying
You might think these dots are shifting,
but in reality, they are static
The system tracks movement by calculating the positional data changes relative to the fixed points
So far, we’ve covered the basic setup,
but there is an additional tracking system that enhances this process
Let's take a look at that
Until now, the tracking systems we discussed focus on positional, or six-axis data
However, for precise rotation, focus adjustments,
and other camera parameters,
an additional Steadif tracking device is used
This device is mounted on the camera rig
and connected to the camera head
It captures the rotation data of the head
and streams it in real time
These data points are then transferred to a dedicated control panel for further adjustments
The additional camera tracking data we discussed earlier is sent here,
allowing operators to fine-tune and manage settings directly from this panel
The devices you’re looking at now
play a crucial role in the ICVFX system
These are the Disguise media servers
The specific hardware model here is the VX4,
which is responsible
for real-time rendering
and high-quality video output
The system handles all aspects of Genlock synchronization
To explain each part briefly,
the Reference setting allows the device to receive sync signals
from external equipment if needed
In this case, since the device is set as the master, it is configured to Internal mode
One of the most critical sections is the Black Burst output setting
This setting sends synchronization signals externally
If we change the format here,
the sync signals of all connected equipment will be updated accordingly
A key feature of this system is the ability to adjust signal formats for each individual output terminal
Currently, the settings indicate a 1080-59.94i format
Additionally, if timecode synchronization is required,
it can be set to LTC or Linear Timecode
The most essential task here is
to monitor the output status panel
At times, when using multiple tools, it is easy to overlook certain details,
so it is crucial to check this setting first before proceeding
Finally, while we have verified the Genlock sync status on other devices,
let’s now check how Genlock synchronization is managed within the media server
At the top of the software interface, there is a Feed tab
Clicking this tab
allows us to see all the settings related to output transmission
In this section, we can confirm that the Refresh Rate is set to 59.94
Additionally,
frame rates can be adjusted according to our needs
There is also a Latency Mode setting
that allows switching between Full Speed and Half Speed,
depending on the project requirements
One of the most important indicators in this interface
is the green status box in the upper-right corner
This indicates whether the Genlock signal
is correctly received by the system
Currently, all indicators are green, meaning that all devices are synchronized nicely
It means that all devices are operating under a single synchronized timing system
Right now, this section appears black,
which indicates that no signal is being sent or received
If this indicator turns red,
it means there is an issue that needs to be addressed
To resolve this,
we need to reapply the Genlock signal
from the Master Device settings that we saw,
and refresh the synchronization settings
Once applied, the system will take approximately 3 to 5 minutes
This process ensures that
the correct synchronization signal is distributed properly
This concludes our ICVFX system overview at VX Studio
So far, we have covered both the theoretical principles
and practical demonstrations of ICVFX system integration
As we have seen, various hardware and software components
must work together in a synchronized manner
Because of this, each component plays a crucial role
We have explored the fundamental concepts and key components of ICVFX
I hope this lecture has helped clarify that ICVFX is much more than simply using an LED wall,
as a highly sophisticated integration of advanced technology and precision-engineered equipment
I hope the knowledge you have gained today will be valuable for your future projects and work
Since ICVFX is a rapidly evolving field,
new technologies and equipment are constantly emerging
encourage you to stay curious,
continue exploring, and accumulate hands-on experience and practical knowledge
With that, let’s summarize today’s lesson and wrap up
Thank you
Basic Concept of VP Traditional Filming Methods: Location shooting, open set filming, chroma background shooting, post-production VFX
VP Technologies ICVFX, MOCAP, PREVIZ, VR, MR, XR, Virtual Camera, SimulCam
Basic Concept of ICVFX VP technology utilizing LED Walls: ICVFX 2D, ICVFX XR
ICVFX Components LED Walls, Image Processor, Camera and Lenses, Media Server or Render PC, Genlock Sync Across All Devices, Camera Tracking System
ICVFX System Overview via Diagram Operating Machine: Controls the entire system, manages network communication and connects to rendering machines
Rendering Machines: Uses Unreal Engine to render 3D environments in real-time and output them to LED screens
Network Switch Hub & Sync Generator: Synchronizes all devices to maintain accurate signal timing, preventing screen tearing or mismatches
LED Image Processor: Adjusts the rendered images to fit the LED screen, ensuring high-quality visual output
Camera and Lens Tracking System: Ensures real-world camera movement and lens data are accurately reflected in the virtual environment
Helps create seamless, realistic scenes
VX Studio On-Site Demonstration LED Wall, Brain Bar, LED Image Processor, Camera Tracking, Media Server, Master Genlock