Reality Advanced Media Server Disguise Utilization(1)
GCC Academy
Hello, I am Jiyong Park
and I will be giving the first lecture on the utilization of Disguise today
In this lecture, we aim to understand various technical concepts
and workflows, such as XR 2D and 2.5D,
and learn how to apply them in practice
To this end, we will focus on
the following four topics
First, we will learn what each workflow,
2D and 2.5D, along with XR
means and in what production environment they are used
We will compare the advantages and limitations of each technology
and understand which is the most suitable
for which situation
Next, we will learn about mapping,
which accurately projects content onto displays or surfaces
We will learn about 2D, 3D
and various types of mapping methods
Third, we will explore the components
necessary to implement XR
and understand the core technologies required to build an XR environment
such as media servers, LED screens, tracking systems,
and rendering engines
Lastly, we will learn the process of
setting up an actual XR environment
We will learn practical settings for connecting and synchronizing
various displays and tracking systems
in various environments
and for stable shooting and rendering
Through these four topics, we will take the time
to understand the latest technologies
related to XR and learn how to use them
to create creative and efficient content
Understanding XR, 2D, 2.5D, and Mapping
First, let's learn about XR
At Disguise, XR
is a general term for live production
combining AR, MR, and VR elements
to provide a fully immersive experience
Using XR, you can provide
an immersive virtual environment to the audience
and convey the functions and design of the product
more realistically
You can provide an experience where the audience
can directly experience the product's appeal
beyond a simple presentation
We've briefly looked at it
but let's learn about the 2D workflow this time
The 2D workflow is a method of capturing
still images or videos, editing them,
and using them as a background for an LED screen
This method lacks depth and parallax effects
but can be used effectively in certain environments
First, the 2D workflow consists of plates
Plates are created based on
on-site photos or videos
and edited using various tools
The completed plates
are used as backgrounds in virtual production
This method is especially useful
when the camera is fixed or requires
a limited range of movement
The main advantage of the 2D workflow is that
it is flexible and scalable
It can be easily applied to various shooting environments
because it can produce high-resolution images
using multiple camera settings
It can also be scaled to meet a specific scene's needs
while maintaining the background's texture and details
This provides high usability in shooting environments
beyond just visual elements
However, the 2D workflow has some limitations
and caveats
The biggest limitation is the limited depth
and lack of parallax effect
meaning that the background does not follow the camera
naturally when it moves
This can be a limitation in scenes
with large camera movements
It may also not be suitable for environments requiring flexibility
as repeated changes on set are impossible
So, what should you pay attention to
when using 2D plates?
The most important thing is the quality of the original plate
To get good results when reshooting in LED volume
it is important to maintain
the highest possible resolution and dynamic range clarity
in the original camera data and background plate
Since the image is displayed on the LED screen in VP
the lack of resolution or noise
can hinder the realism
Therefore, it is recommended to use a camera with at least 8K resolution
or higher and shoot in the highest possible bitrate
and log or raw format
In conclusion, 2D plates are a powerful tool for implementing cost-effective
and realistic backgrounds in VP environments
However, it is important to maximize this method's advantages
while being aware of its clear disadvantages
In environments where the camera is fixed
r limited movement is required
2D plates may be a more efficient
and convenient choice than a 3D workflow
The first case we introduce is
the 2D plate in the movie Exhuma
This scene was used as a background for the hotel window
They filmed the night view of the city
and installed LEDs outside the hotel set
in the studio
2D plates can be used effectively in this way
with a relatively fixed camera position
bringing about natural harmony
between the background and the characters
In the case of a luxury hotel like Exhuma
it is difficult to film on location
and even if we do film on location
it is a high-rise building where we cannot film outside the window
so this is a very suitable solution
for special location situations like this
Since it provides an accurate background image according to the camera angle
the audience can feel immersed
as if looking out the window
The next work we worked on was
Netflix's D.P. Season 2
Here, we worked on a 2D train driving scene
and in that scene, the actors acted on the train set
and the background seen outside the train
was implemented with 2D plates
As you can see from the series, the train scene is very intense
the route is complex, and there are many characters
so you can see that
2D driving is an excellent choice
We were able to freely film the actors' actions
in a safer environment
and we were able to film various repetitive scenes without time constraints
This train scene was planned through
testing in advance
We used the set and LED that Disguise had implemented
in advance to do techbiz
Here, tech biz
is the process of planning such complex filming
and you can think of it as a technical simulation in advance
by combining virtual elements
with real objects and equipment
Techbiz is an important selection process
that is necessary before filming
as well as camera movement, placement, and lens selection
The difference between a team that actually performed techbiz
and a team that did not is huge on-site
Techbiz is an essential process that minimizes on-site work time
and prepares for costs
and various variables
So far, we have learned about
2D workflow and its uses
2D is a powerful tool for efficiently implementing
fixed backgrounds
and simple visual elements
However, in cases where complex scenes, camera movements,
and higher immersion are required,
2D alone has limitations
The 2.5D workflow emerged
to solve these limitations
The 2.5D workflow is a technology that bridges the gap
between existing 2D video plates and fully 3D-generated scenes
In simple terms, it implements parallax effects
by stacking layers on images or video plates
and adding depth and shape to them
Through this process, more realistic scenes
can be created quickly and efficiently
So, what are the main advantages
of a 2.5D workflow?
Parallax effects and depth perception
A 2.5D workflow is ideal for scenes
requiring small camera movements and a little depth
It has the great advantage of
creating realistic scenes without sacrificing photorealism
When the camera moves, the depth of the background is naturally expressed
allowing the audience to experience a more immersive environment
Second, it saves time and budget
The 2.5D workflow uses layered plates
to create a sense of depth
It can create immersive environments
without the need for long pre-production times
and save on the cost of
3D asset production or VADs
This allows small-scale and large-scale productions
to enjoy the benefits of
virtual production without budget constraints
Third, it increases the flexibility of virtual production
The 2.5D workflow extends the flexibility
that virtual production already offers
It makes it easier to create realistic environments
by adding additional depth to the scene and creating
intermediate points that differentiate it from the background
In particular, it gives directors and cinematographers
more possibilities to explore what else
they can add to the scene within budget
In conclusion, the 2.5D workflow
can be seen as an efficient and flexible solution
for creating realistic scenes and immersive environments
within budget and time constraints
The 2.5D workflow can be widely applied
in a variety of production environments
The most common example is set extension in films and TV dramas
You can easily expand
large-scale backgrounds or environments
that are physically difficult to produce
using image plates and depth layers
This can greatly reduce production costs and time
If you have learned about 2D, 2.5D, and 3D
let's take a moment to briefly understand
how content is mapped in Disguise
First of all, what is Mapping?
Mapping is an important characteristic used in accurately conveying
content as an expression of a specific screen or stage
You can convey the content of a single video layer
to single or multiple screens
or map the content to screens
with irregular shapes and resolutions
Mapping allows you to combine digital content and physical space
to provide a more immersive
and multidimensional visual experience
Next, we will look at the various types of
Disguise mapping and the characteristics of each type
Mapping is largely divided into
2D, 3D, and geometry categories
Each category is selected based on the nature of the content and the target of Mapping
and can be divided into static or dynamic Mapping,
2D, and 3D
Static Mapping is a method suitable for
projecting fixed images or videos on a screen
and is useful for settings with little environmental change
Dynamic Mapping is used when mapping content
that changes in real-time
and is widely used in live events
The standard for dividing 2D and 3D Mapping is that
2D Mapping is suitable for placing content on a flat screen
and is mainly used on rectangular displays such as LED walls
On the other hand, 3D Mapping means
Mapping complex-shaped surfaces, objects, or upper spaces
Geometric Mapping means
Mapping to a fan or cylindrical surface,
as in the case of Spherical or Cylindrical Mapping
Parallel, Perspective and Radial Mapping
means Mapping by using multiple surfaces as a single canvas
or applying perspective to project content
Let's look at some of the more
common mapping methods
The first is direct mapping
Direct mapping is one of the static mapping methods
and the most basic form of mapping in Disguise
It is a mapping method that is automatically set when creating a screen
Since the content is directly applied to the texture coordinates of the object
the content is fixed to
the corresponding screen even if the object moves
When the content is initially loaded, the Disguise software
sets the default value to Clip to Canvas
so the aspect ratio is fixed and displayed
However, suppose the content's aspect ratio differs from the screen's
In that case, the operator can crop, fit, expand,
or apply pixel-perfect to the content
on a pixel-by-pixel basis
The second is feed mapping
Feed mapping is a very useful and flexible mapping method
It allows you to route specific areas
to specific screens and surfaces
The main advantage of feed mapping is that it allows you to supply content
to multiple screens simultaneously using a single content source
This is especially useful for
show programming that uses multiple screens
and it also helps improve server performance
by reducing the number of layers required
to supply content to multiple screens simultaneously
The third is parallel mapping
Parallel mapping works by
geometrically projecting content onto the scene
almost like a projector or other illuminant
shoots the content virtually
Unlike other mapping methods, parallel mapping projects the content
onto the screen rather than directly attaching it to UVs
This allows the content
to dynamically update as its position
in 3D space changes
This means that, unlike direct or feed mapping
the content is not directly attached to the surface
so the content will look different
when the viewpoint changes or objects move
This is advantageous in environments where the physical screen arrangement
is not fixed, such as the exterior of a car
So far, we have looked at 2D, 3D, and 2.5D concepts
and the various mapping methods
The important question is
in what situations should you choose between 2D, 3D
between 2D, 3D and 2.5D?
Each workflow and mapping method has advantages and disadvantages
and choosing the method that suits
your project's purpose and requirements is very important
2D is suitable when you need a fixed background
or simple visual elements
Its main advantages are that it is cost-effective and takes less time to produce
and post-production is simpler than 3D
However, its limitations include a lack of depth
and limited camera movement
Therefore, it is suitable for static or simple scenes
but may have limitations in complex visual production
3D is often used in large-scale projects
such as movies and dramas,
and it provides more immersive visual effects by integrating actual physical sets
and digital environments in real-time
The biggest advantage is that it can implement realistic screens
through real-time camera tracking
Since the digital background reacts dynamically
according to the camera's movement
the director and staff can immediately check
and modify the scene
However, this method requires higher production costs
and complex setup processes than 2D
In addition, it may be difficult to apply to small-scale projects
because it requires a high-performance hardware system
I think you can think of 2.5D
as a method between 2D and 3D, as I mentioned
It is a middle ground
that provides cost and time efficiency while taking advantage of each
ou can use multiple layers to work
in 2D while achieving some 3D effects
If you clearly understand and utilize these advantages and disadvantages well,
you can achieve the best results while producing efficiently
Components and Workflow of XR
Let's go back to XR
As I mentioned, XR is an extended reality concept
that includes virtual, augmented,
and mixed reality technologies
XR is an innovative technology
that blurs the boundaries between virtual and real by combining
real-time virtual environments with physical elements such as LED screens
This technology is used in various industries
such as music videos, live broadcasts, and film production
providing viewers
with amazing visual experiences
Now, let's look at how
the XR stage is structured
First, let's explain the camera view
The camera view shows how the XR environment
is ultimately rendered
Through this view, the director and production team
can check the final visual results
in real-time and immediately reflect on any necessary adjustments
Therefore, the camera view plays
an important role in production
Next is the LED stage, the central element of the XR stage
The LED stage is the core technology of the XR environment
where the virtual background is physically expressed through the LED panel
Actors and hosts can act or proceed naturally
while viewing the virtual background through the LED screen
And the outside of the LED stage
is expressed as a virtual set extension
This extension allows the virtual environment
to continue beyond the LED panels
giving the audience a sense of infinite space
and allowing for a production that transcends
the constraints of actual physical space
Finally, the floor
The floor combines physical elements and LED panels
to help actors and presenters interact
naturally with the XR environment
The floor with LED panels enhances lighting and reflection effects
to create a more realistic environment
To implement immersive content
and visual effects in an XR studio
various components must be organically connected
Here, we will look at five key elements
essential for XR implementation
First, the LED screen
The LED screen is a display device
that forms the physical foundation of XR
and displays the virtual background in real-time
Second, the camera tracking system
The camera tracking system records
the camera's position, angle, and movement data in real-time
and transmits it to other systems
Third, the media server
The media server acts as the hub
of the entire XR system
Fourth, the Genlock Sync
The Genlock Sync maintains frame synchronization
between all devices and systems
Lastly, the real-time content rendering engine
The real-time rendering engine provides high-resolution 3D graphics
and physically-based rendering in real-time
and is a key technology for completing visual elements in XR
In this way, the components of XR play their roles
while organically connecting to create an immersive environment
where the virtual and the real are naturally blended
Let's take a closer look at each of these elements
One of the most important components of an XR stage
is the LED screen
The LED screen physically implements the virtual environment
allowing actors and props to interact
with the virtual background in real-time
The core of an XR stage is a structure that includes walls,
floors, and sometimes even ceilings made of LED panels
This setup is very important
Because the resolution and LED pixel pitch
directly affect the visual quality
and clarity of the digital content
In XR, every detail matters
The pixel density of these LED panels
ensures that the virtual environment looks smooth
and realistic even when shot up close with a camera
LED screens can be installed on walls, floors,
and even in dome shapes,
allowing for flexible background implementation in any environment
In particular, floor LEDs are the most important element of XR
and floor LEDs are essential
for XR to become Extended Reality
or spatial expansion, as its name suggests
The next core component of XR that we will introduce is
the camera tracking system
This technology is essential for
synchronizing the virtual background with the actual camera movement in an XR environment
The camera tracking system
tracks the position and movement angle of the camera in real-time
allowing the virtual background to
automatically adjust to the camera’s viewpoint
This makes the viewer feel like the camera
is moving in real space
As the camera moves, the virtual environment displayed on the LED screen
also changes accurately according to the camera’s viewpoint
This makes it appear that the virtual background and the real person
are in a consistent space
So, what technologies are used
in these camera tracking systems?
Representative camera tracking systems include
Skype, Ncam,
and the Optitrack mentioned earlier
These devices precisely track the position and movement
of the camera using high-precision sensors
The camera tracking data is combined with Disguise software
and reflected in the virtual environment in real-time
It is necessary to add and activate a driver
that matches the Disguise tracking system
create a position receiver
and connect it to the tracking system
Camera tracking systems are essential for all types of XR production,
such as movies, advertisements, and live events
They are a technology that perfectly merges
the virtual and real in the XR environment
Next, we will learn more about the real-time rendering engine
that plays a key role in the Disguise XR environment
The rendering engine is the core technology
that determines the quality of XR content and real-time responsiveness
A real-time rendering engine is a program like Unreal Engine or Unity
that creates and renders 3D content in real-time
and provides immediate visual feedback
on LED screens and virtual environments
Epic Games' Unreal Engine is a prime example
It allows for a high level of realism
and dynamic environment control
You can build environments composed of 3D
and use a studio space as a multi-purpose space
to continuously create
new content in the same space
You can render scenes as the camera sees them
maximize realism
and create perspectives for the scenes
to provide deeper and more vivid screens
The real-time rendering engine also supports
integrated workflows with
Notch, Unity, and Unreal tools
The camera can move freely in the 3D environment
and is not tied to fixed shots
or limited camera positions
Let’s take a look at the XR workflow
Networking is a key element in any XR projection
After all, the data flow between the server and the rendering engine
heavily depends on the network infrastructure
The network must handle the bandwidth
of all the generated and transmitted data
So, what types of networks do we need
in an XR system?
To build an effective XR system
we need the following networks
First is the media server network, d3Net
d3Net handles communication between all the servers
in the XR system
The media server supports fast and reliable data transmission
Next is the metadata and video Ethernet transmission
This is called the render stream
which is the way to communicate with third-party render engines,
such as Unreal Engine
This allows us to transmit metadata
and video data between the VX and RX
For high-quality video streams
we need to transmit uncompressed data
and support 10-bit transmission
to maintain color accuracy and detail
We also need to utilize a 25Gbps network
to quickly transmit large amounts of data
The system must also track camera movement data
through automation and camera tracking networks
to provide real-time synchronization
The last, content management network
provides the connectivity needed
to upload, download, and manage content
It is used to retrieve media files
from external sources for downloading
Let’s look at the multiple networks required for an XR system
Looking at the VX series
the system is equipped with multiple NICs
These NICs enable a variety of bandwidth activities
The VX4+ has one 1Gb port, two 10Gb ports
and two 100Gb ports
and this configuration supports real-time rendering
and high-speed data transfer
By separating the various data types,
such as communication, generation, content streaming, or camera tracking
each channel can operate efficiently without congestion
optimizing system performance
So, let’s organize the system network configuration
diagram of the configurations above
First, the Disguise equipment
has the VX and RX systems in the center
The VX is connected to the LED processor
to output high-quality content
and the RX receives content from a rendering engine
such as Unreal Engine and transmits it to the VX
This is connected via d3Net
a dedicated network between Disguise equipment
On the left are the video cameras and the tracking system
Each camera captures real-time data
and the tracking system transmits the camera’s position
and movement data to the VX4 based on this data
The entire system is also perfectly synchronized
via the Genlock system
Next is the network
The 10Gb network switch and the 25Gb Mellanox switch
support high-capacity transmission
to process the render stream and KVM signals
On the right are the LED processor, the display
and output-related parts
This processor receives the data transmitted from the VX4
and accurately displays the content on the LED screen
We’ve covered the network
but a good system starts with a solid plan
As with any VP shoot
the system administrator
plays an important role in an XR production
They are responsible for designing and planning the overall system
configuring the individual networks, and ensuring
smooth communication between the various devices
We’ve covered some of the important
individual components of XR so far
This time, I’ll explain
how these elements are organically
linked to the workflow
Everything starts in the Disguise Extended Reality Controller
Disguise Designer software
sets up the exact virtual environment
through camera and lens calibration
spatial and color mapping, etc.
This includes the Disguise 3D simulation
and video mapping server
so that each element is precisely synchronized
The second step is real-time graphics generation
This process uses
a rendering engine like Unreal Engine
and the virtual environment is output to the LED screen
The RenderStream plugin creates
the AR content in real time
and this data is rendered in real time
along with the camera tracking data
All of this content is then fed to the LED wall
This is where the LEDs play a key role
The lighting control system
also synchronizes the light from the LED screen with the physical lighting
to create a more immersive environment
The final step is Vision Mixer
This is where the final image composition takes place
The AR content and lens data are combined
to create a perfectly integrated screen with a real camera
and virtual background
This result is ready to be broadcast or streamed
directly to streaming platforms
Overall, the Extended Reality Workflow
integrates all XR components in real-time
streamlining the process from pre-production, real-time lighting
and post-production
to ensure high quality while saving time and money
Let’s examine the core elements of 3D stage calibration
in the Disguise XR workflow
Accurate calibration ensures that
the real and virtual worlds are aligned seamlessly
enabling immersive productions
There are two main types of calibration
Spatial calibration and lens calibration
Spatial calibration involves World Alignment
which is
the process of accurately aligning the real world
such as the actual stage or camera position, with the Unreal virtual world
It accurately interprets tracking data
to match the coordinate system
so that the camera moves in physical space
and is reflected in the Unreal virtual environment
It adjusts the virtual content to blend naturally
with the real environment without distortion
In other words, spatial calibration
is the foundation for aligning the real and virtual spaces
Lens calibration profiles the optical characteristics
of a real camera lens
to compute a unique profile for the lens
This profile is essential for matching the distortion
and field of view of the rendered virtual extended set
with the real camera lens
The virtual 3D models inside Disguise must be 100% accurate
as even a small error can ruin the immersion
and disrupt the audience’s immersion
if the physical stage
and virtual model do not match
Finally, there are countless factors to consider
when designing an actual XR system
but here are a few examples to keep in mind
If you use different types of LED processors
each LED processor may have different color mapping, gamma correction
and brightness levels
which increases the likelihood of color differences
and each processor requires its settings and calibration
This can ultimately lead to longer installation and setup times
If the refresh rate is not high enough
flickering, or screen flickering, can occur
If the signal formats do not match,
when devices using different video signal formats
are connected
the signal may not be recognized
or transmitted
If the signal formats between devices do not match
downscaling or upscaling may occur
which may cause image quality degradation and color distortion
If the signal formats do not match, signal conversion is required
which may cause additional latency
In addition, audio and video may be out of sync
I hope you will continue to explore with curiosity
and gain practical experience
and knowledge necessary for the field
This time, as the first time using Disguise
we aimed to understand
various technical concepts and workflows such as XR, 2D,
and 2.5D and learn how to apply them in practice
I hope this was a time for you to understand the latest technologies in XR
and learn more creative
and efficient ways to create content
for your works
Then, let's finish by reviewing
what we learned this time
Thank you
Understanding XR, 2D, 2.5D, and Mapping XR (Extended Reality)
A general term for live production that combines AR (Augmented Reality), MR (Mixed Reality), and VR (Virtual Reality) elements to provide a fully immersive experience
2D Workflow Capture and edit still images or video to use as LED screen backgrounds
Advantages and limitations of 2D Workflow Excellent flexibility and scalability Scaling to suit the needs of a specific scene while maintaining the texture and detail of the background
Limited depth, no parallax effect Impossible to make repeated changes on set
2.5D Workflow A method of implementing the Parallax Effect by stacking layers of image or video plates to add depth and form
Advantages of 2.5D Workflow Provides Parallax Effect and Depth Saves time and budget Expands flexibility of virtual production
Mapping Major properties used in the process of accurately delivering content to specific screens and stage expressions
Representative mapping methods Direct Mapping: One of the Static Mappings, Disguise's basic mapping method that is automatically set when creating a screen
Feed Mapping: A useful and flexible method that allows routing specific areas to specific screens and expressions
Parallel Mapping: A method that geometrically projects content onto a scene, acting as if the content were being filmed virtually
Components and Workflow of XR Components of XR Camera View: Shows the final rendering result of the XR environment
LED Stage: Core technology of the XR environment, where the virtual background is physically represented through LED panels
Floor: Combining physical elements and LED panels to help actors and presenters interact naturally with the XR environment
XR Implementation Essentials LED Screens(Stage) Camera Tracking System
Media Server(disguise) Genlock Sync Real-Time Content Rendering Engine
Network required for building XR system Media server network (d3Net) Metadata and video Ethernet transmission (RenderStream) Content management network
Types of 3D stage correction Spatial correction: Spatial correction including World Alignment Lens correction: Profiling the optical characteristics of real camera lenses and calculating the lens-specific profile
XR Workflow Extended Reality Controller Real-time graphic generation LED Wall (Stage) Vision Mixer