Realistic
Realistic Advanced Cinematic Videos for ICVFX
GCC Academy
Hello
I am Kim Saet-byeol, and I am in charge of the realistic media content lecture
This lecture will cover icvfx in-camera visual effects
combining cinematic video production with cutting-edge technology
Cinematic Videos and ICVFX
ICVFX technology allows you to complete the final video
by applying visual effects in real-time on the shooting site
unlike the existing post-production method
This technology is used in various visual media
such as movies, advertisements, and media art
and opens up new possibilities for creators
Cinematic video production techniques, key elements of ICVFX
a solution supporting various screen ratios
Let's introduce cinematic video transmission and rendering
and see how to achieve creative and immersive results
First, cinematic video is
a technique that emphasizes the visual senses, like a movie
and highlights dramatic elements
in scene composition and visual style
Cinematic video
uses various visual techniques to immerse the audience
and induce emotional responses
First, let's look at the elements of cinematic images
The first is lighting
Lighting is a very important element in cinematic images
and it determines emotions and moods through the direction, intensity, and color tones of light
For example, soft light gives a warm and intimate feeling
while sharp and dark lighting adds tension and mystery
In media, LED lights or projections change
specific colors or moods in real-time
In cinematic images, a single color tone or a combination of specific colors
can set the overall mood of the work
The second is frame and composition
The context and emotions of the story can be conveyed
through the position of characters or objects, the angle of the camera, and the composition
For example, there are examples of emphasizing the environment in which the main character is located
by placing it in the center of the screen
and providing a wide background or enhancing the character's sense of authority through a low angle
By adjusting the depth of field
you can create different scenes where the focus is on, and the blurry parts are different
It is used to focus the gaze on a specific object or person
or to differentiate the background and the subject
The third is camera movement and angle
Various camera movement methods
such as pan, tilt, zoom, and tracking
can add liveliness to the scene
In a scene where a character is moving
a tracking shot that moves with the character can increase immersion
and a slow zoom-in shot can convey tension
High Angle, Low Angle, and Overhead Shot
can visually convey the scene's context
Overhead Shot
which means a shot looking down from above
In the case of overhead shots like this
a low-angle shot looking up from below
effectively expresses a sense of authority and grandeur
The fourth sound and music cannot be left out in cinematic images
To add a cinematic feel, music that matches the emotion and atmosphere should be used
For example, lyrical music would be effective for emotional scenes
and urgent music would be effective for thriller scenes
Sound effects enhance the realism of cinematic images
For example, environmental noises such as wind and footsteps
commonly referred to as SFX
can draw the audience into the scene
and enhance immersion by allowing them
to experience visuals and sounds together
Fifth is editing and rhythm
Fast cut editing can induce urgency
while slow editing can induce calmness and empathy
Transition effects, fade ins, and fade outs
can help scenes connect naturally to each other, increasing immersion
The tempo should be adjusted to fit the story through the entire image's flow and the scenes' length
Rhythmic editing can maintain focus and tension
while slow development can leave a lingering impression of the work
Next is storytelling
It is the most essential element of a video
Even without dialogue or text
you can convey a message through scene composition
and help the audience understand the story through visual elements
To do this, you can use metaphorical scenes and symbols that intuitively convey themes or emotions
Media art can implement cinematic images by combining
cinematic elements with installations or projections
You can create a video environment where the audience
intentionally participates in the work or reacts in real time
You can combine interactive technology with these cinematic images
to experience new sensations
The aspect ratio is very important in cinematic images
The aspect ratio here refers to the width and height ratio of the image
Each aspect ratio creates a specific emotion and atmosphere
Through this aspect ratio, the viewer has a different visual experience
I would like to introduce the most commonly used aspect ratios
16:9 wide screen
is the basic aspect ratio of TVs and most digital devices
This aspect ratio allows for balanced visual expression
and is widely used in various genres
Next is 21:9 Cinema Scope
It is a screen ratio often used in theater movies
and can add grandeur to the scene because it provides a wide field of view
It is also a ratio that can increase the sense of scale and immersion in cinematic images
Next is 4:3, a screen ratio that can express a classic or retro vibe
It is a screen ratio that is often used in TV broadcasts
It has the effect of increasing emotional concentration and stimulating analog feelings
Next, I would like to introduce ICVFX in-camera visual effects
ICVFX stands for in-camera visual effects
It is a technology that uses real-time effects
during filming to increase visual immersion in movies or media works
It mainly uses LED screens, real-time rendering software
such as Unreal Engine and Unity
and camera tracking systems
to show the desired background or environment during filming directly
Unlike VFX, which adds effects to the filmed screen later
ICVFX is characterized by obtaining a completed scene right during filming
ICVFX uses a stage surrounded by LED panels
called an LED volume
This large LED screen covers the entire wall and ceiling
providing a background that changes in real-time as the actor moves
The LED volume allows for real-time 3D background projection
allowing the actor and background to blend in naturally and be filmed
This reduces the need for background compositing in post-production
saving cost and time
You can often see this in places called XR studios
The core of ICVFX is real-time rendering
Game engines such as Unreal Engine or Unity are mainly used
Complex 3D environments and detailed backgrounds are quickly rendered and projected onto LED screens
The advantage of real-time rendering here is that
the background is adjusted immediately whenever the camera position and angle change
allowing for a natural parallax effect
or parallax
This makes the film's characters and objects appear to exist in space
Camera tracking is also essential in ICVFX
The tracking system monitors the camera's movement in real-time
and adjusts the background to change according to the viewpoint of the moving camera
When the camera moves, the background is synchronized to the viewpoint
making the video look more natural
This allows the background to appear as if it were a real place
when the camera follows the actor or object
Since the LED volume can also act as an actual light source
it creates natural light and shadows on the actor or prop
This lighting effect allows the light to naturally reflect on the character's skin or the object's surface
making for a more realistic production
This real-time lighting effect
reduces color correction or lighting modification in post-production
and more precise and natural scenes are completed
This ICVFX allows
the actor to act in a space that looks like a real environment on a large LED screen
which increases immersion
Another advantage is that the actor and the filming staff
can check the completed feel of the scene in real-time
In addition, this ICVFX
can greatly reduce the time needed to compile the 3D background in post-production
Existing VFX can take a long time to finish the background composition
With ICVFX, you can complete the background at the same time as shooting
saving time and cost
Also, one of the biggest advantages is that
you can set the desired time and weather conditions
anywhere through LED volume to shoot with
consistent lighting and environment regardless of the weather
Since there are no restrictions on location
you don’t have to go to overseas locations
Lastly, you can immediately see the changes on the shooting site
when the background moves or changes through LED volume
This allows you to adjust and provide feedback
in real time according to the intention and atmosphere of the work
making shooting more flexible
In media art, ICVFX can be applied to
realistic installations or
interactive art that encourages audience participation
In an exhibition space using LED volume
you can create an interactive environment where the background changes as the audience moves
or you can make the audience become a part of the media artwork and have an immersive experience
This ICVFX technology provides a realistic sense of reality
immersing the audience as if they were in a real space
Solutions for Adapting to Various Aspect Ratios
Unlike the traditional post-production method
ICVFX is a technology that allows you to apply visual effects in real time on set
to complete the final video
In the case of Unreal, it has a very strong feature of
real-time rendering scalability for ICVFX
It is also widely used in movies and media
Because the environment for preparing assets for multiple aspect ratios is optimized
it is very convenient
You can adjust the composition of major visual elements
to fit the aspect ratio of all objects through frame-safe or flexible framing technology
Also, using virtual cameras
you can also simulate output for various display formats
It can be produced so that it can adapt to dynamic content
You can also automate scaling or framing logic using Blueprints
In particular, you can see cinematic scenes with dynamic camera movements
or scenes adjusted to 16:9, 9:16
or ultra-wide formats
When preparing media art or content
there are many cases where you have to respond to displays with very unique ratios
Therefore, you can apply film principles to multi-aspect ratio displays
to express them cinematically
First, since the nDisplay plugin is required, open the Unreal Editor
Up to version 5.5 has been released
and I am using version 5.4
I want to bring in an empty project from the tab called Film and TV Live Events
Set the name as In-Camera Project and click the Create button
You can proceed with various names of your own
First, when the project opens, the main level is open
and there is currently nothing
Go to the plugin window and check if nDisplay is installed
If it is not installed, please restart the program
to install nDisplay
Looking at the window, you will see an in-camera VFX in the level editor tab called Virtual Production
If you open it, it is not activated because there is no nDisplay actor yet
This
after creating the nDisplay actor
I will return to the in-camera VFX window and check this part
First, create the folder you want
and then I will create a basic folder
and organize the assets related to nDisplay
Enter the folder, right-click
select nDisplay Preferences
and proceed with the new environment settings
The nDisplay name
is set to Incamera_1
If you double-click and enter the nDisplay screen
you can see the following
Basically, one nDisplay screen is set
and the viewpoint
cluster, and output mapping parts are not yet shown
so I will set those parts as well
You can copy the nDisplay screen with Ctrl+C and V
or adjust the corresponding view in the screen size section
If you change the value
you can create the screen size of the space I map
This screen is locked
so please open it
It may be a very long screen
with a unique horizontal shape, such as vertical or horizontal
Let's create a screen like this
and add a cluster
Add a cluster node and set it to FHD
but we will work on this part
by increasing it to 4K and Ultra HD
When you add it, a viewpoint is created in the output mapping section in the middle
and this viewpoint is currently not connected
so let's look at mapping this viewpoint to the aspect ratio section we created
To connect the created screen and the viewpoint
click viewpoint 0 first
You can open the projection policy section as a drop-down
The type is simple
If you change this part to mesh
and apply the name of this mesh, nDisplay screen
the corresponding viewpoint will be depicted
You can add a viewpoint to add a second screen
It's called a viewport
Add another viewport, and it'll look like this
If you connect this viewport to the mesh next to it, it'll look like this
It looks like you're looking at the same space, but it seems like a completely different screen
If you put it in, two viewports will be set according to the viewpoint
If you click compile and come out
the corresponding nDisplay will be created here
If you drag and drop it here, you can see how it looks
Let's set the screen to be visible through Quixel Bridge
Let's go into Quixel Bridge
add some simple assets, and place them on the screen
I'll do it at medium quality
I downloaded some assets from Quixel Bridge
and added them to the corresponding content
Since no map's visible here, I'll put it in a little
and set it up
You can see it to some extent
If you look, what's located on the screen will be visible on the display
If you put it like this, you can see it
So, as you can see, the shape displayed on the nDisplay screen
and the shape placed in reality is slightly different and distorted
If you put the screen that is installed like this back into the in-camera vpx of virtual production
and place an actor on the nDisplay screen
it will look like this
So, the same scene can be linked and shown
For example, I will put it like this and place a corresponding flower
Can you see that the added flower appears on the display screen?
So, even if it moves
to the opposite side
you can see this environment linked on the screen in the in-camera VFX
Also, if you move this nDisplay screen
the ratio on the screen changes depending on the camera's movement
These various aspect ratios
and various types of environments can be linked as a whole
This can be used through the nDisplay screen and in-camera VFX tool
and the corresponding flower can be seen like this
Let's move from a simple environment to a more complex environment
and create a screen with a new aspect ratio
When using a map, since you will be bringing in various units
each of those units has different units from Blender, C4D, Maya, etc.
so when you finally bring it in from one editor
we recommend that you import and bring in
all of the size, ratio, visible direction, and material texture
I brought a traditional Korean map
I will create a new nDisplay and put it in
This time, I will set it to a square
4-sided display
When you come in, it is processed like this
The screen is the same
You can add a screen here
and bring it in with a static mesh
but today, I will try adding a screen and putting it in
Adjust the position so that the nDisplay screen and the corners touch by giving a rotation value
Let's make 4 sides by entering a number
Copy the nDisplay screen, set the rotation value to 0,0,180, and adjust the position to face each other
As you can see, you can see where the corresponding side is invested on one side
You need to design this part exactly to see it well
Copy nDisplay screen 1 and adjust the position to face each other
I placed it like this until the last screen
and moved all the screens slightly
so that they could be viewed from the center based on the viewpoint
I added a cluster in the same way
and placed it like this
Here is the viewpoint
You can set the screen for this viewpoint
You can also reduce this viewpoint to adjust the ratio
Right now, the size is set like this
so I'll proceed with the ratio of dividing it into 4
I'll adjust one viewpoint to a 2:1 ratio
and here to the same 2:1 ratio
Then, if you add a viewpoint like this
you'll see two
You can see it like this on the second screen
But I will change
the size of this host, the node itself
I'll change it to the ratio we're making
If you enter a number like this, you can respond to various aspect ratios
I have to continue
so I'll put this guy on this screen
and agree to 2000 to 1000
Yes, it's connected
It's set to 8000 to 1080, so I adjusted the viewpoint to fit it up to 1000
Then, if you save this file
and bring this with you
you can see the 360-degree scene shown within the corresponding viewpoint
and check it in the same window as Virtual Production in Camera VFX
So, the possibilities for what kind of scenes can be created
with this nDisplay screen I created are endless and free
And because you can check it before mapping
when it is finally broadcast through this in-camera VFX
you can finish the design like this
Cinematic Video Output and Rendering
Next, I will lecture on rendering and broadcasting
Broadcasting the cinematic video you created is also a process
In Unreal, you can work very flexibly
using Cinematic Studio
Since it is a real-time 3D production tool
you can render movie-quality graphics
and see them immediately
greatly reducing the rendering time
So, since real-time illumination, room and ultra-high-resolution meshes are possible
the cinematic video looks like a movie slate
at the top called Level Sequence
If you add this friend
you can create a level sequence
and open the sequence by adding an appropriate name
You can add shots
, and since we will output it to nDisplay
we will leave it like this
give animation to this nDisplay itself
and do a simple rendering
Add the transform as a component in nDisplay
and output about 150 frames
so that the transform moves
So
if you move like this
a movement mechanism is created
It moves in an animated state
So
let's finish it by giving it a rotation like this
Let's mark a point like this and render it
There are Movie Scene Capture and Movie Render Queue for rendering
but we mainly use the more powerful Movie Render Queue
and it is important to set the path and settings in it
We will not proceed with Deferred rendering
but will output it as a JPG sequence
hrough nDisplay and nDisplay rendering
You must not forget to include the nDisplay root actor
It would be good to output other parts as they are
and it is important to put the directory in a known place
Accept and press Local
to proceed with rendering
There are a lot of features within the sequencer
Since we can see various features commonly used in movie editing programs
such as keyframes, resolution, frame rates, etc.
they are all introduced
in various ways in the Unreal Engine documentation provided by Unreal
Some functions include taking keyframes, adjusting or customizing the FPS sequence display speed
and creating natural movements through curves within the sequencer
Today, we showed you how to set up using the movie render queue
The images output this way
are then moved back to Premiere or After Effects
to be sent out as final videos
You can import the sequences to be edited
adjust the video's color, add effects, etc.,
through Premiere and the encoder
Open the output image sequence with it selected
and if you look at the corresponding file
you will see a video of about 5 seconds with the nDisplay rising
Export this back to the media encoder, finish editing
and then export it to the Media Encoder
The final display and transmission methods are very different
Please check the audio
and codec sections on-site
before completing the final extraction with the encoder
If you bring it to the encoder, there are various options in the preset
and you should check the requirements on-site
and output it according to your expression method, such as maximum rendering
You can proceed with mapping the completed video
on-site where it is set
We mapped a total of 4 sides
so all 4 directions are currently output
and each viewport is output
You can take this to a mapping program
and simulate it in a virtual environment
and you can also output it by connecting it to the actual field
but you can also output it as a separate file
You can match the total length like this and see it all at once
and you can proceed while checking the distortion part
when it is mapped
The cinematic video and ICVFX introduced today
are the most notable and leading areas in realistic media
I hope you can effectively produce more immersive
and diverse expressions through this technique
That concludes this lecture
Cinematic Imagery and ICVFX Cinematic Imagery A technique that emphasizes the visual sense of a film and brings out the dramatic elements of scene composition and visual style
Major elements of cinematic imagery: lighting, framing and composition, camera movement and angles, sound and music, editing and rhythm, storytelling
Aspect ratio The ratio of the width and height of the image Types and characteristics of representative aspect ratios 16:9 The basic aspect ratio of widescreen TVs and digital devices, allowing for balanced visual expression and used in various genres
21:9 Cinema Scope: Aspect ratio often used in theater movies, a ratio that can increase scale and immersion in cinematic images
4:3 Classic retro sensibility: Has the effect of stimulating analog sensibility, which was often used in TV broadcasts in the past
ICVFX A technology that increases visual immersion in movies and media works by implementing effects in real time during shooting
Key elements of ICVFX: LED volume, real-time 3D rendering software, camera tracking system, lighting & reflection
Advantages of ICVFX: Increased sense of presence and immersion, time and cost savings, shooting without location and weather restrictions, providing on-site interactive experience
ICVFX in Media Art Realistic installations and interactive art that encourage audience participation Interactive environments where the background changes as you move in the LED volume exhibition space
Immersive experience environments where the audience becomes part of the media art work
Solutions that can support various screen ratios Unreal Engine & Unity Very powerful real-time rendering scalability for ICVFX Optimized environment for multiple aspect ratios
Flexible framing technology allows major visual elements to be configured to fit the aspect ratio of all objects Various display output simulations using virtual cameras are possible
It can be produced to adapt to dynamic content You can automate scaling and framing logic using Blueprints You can create cinematic scenes with dynamic camera movements
You can adjust the screen to 16:9, 9:16, ultra-wide, etc. You can create cinematic expressions that apply film principles to multiple aspect ratio displays
ICVFX settings linked to nDisplay environment In Unreal, select an empty project in the Movie & TV Live Event tab, set a name, and click the Create button In Edit-Plugin, check the installation of nDisplay
Create a new folder in the Content folder and open the nDisplay screen Copy the nDisplay screen and set the nDisplay screen to be long vertically and nDisplay screen 1 to be long horizontally
Select Add New Cluster Node and set the preset to Ultra HD Change the type of the VP_0 Detail Window Projection Policy to Mesh and apply the nDisplay screen
Select Add New Viewport and set the preset to Ultra HD Change the type of the VP_1 Detail Window Projection Policy to Mesh and apply nDisplay screen 1
Compile and save, then drag and drop InCamera_1 nDisplay to the world Select Window-Quixel Bridge and set it to Medium Quality, then download the necessary assets and place them in the world
Select Window-Virtual Production-InCamera VFX and apply InCamera_1 to the AndDisplay environment
Cinematic video transmission and rendering Click the slate icon, select Add Level Sequence, set the name and save
Drag and add nDisplay_4 to the sequencer, then click the + button on the right to Add Transform Track
Add a transform keyframe to the start point, then move the marker to frame 150 Select the camera position, then add a keyframe to the position track, and rotate the camera, then add a keyframe to the rotation track
Select the movie render queue in the shape of a slate icon in the sequencer window, then add nDisplay and nDisplay Rendering Apply nDisplay_4 to the DC root actor of nDisplay and click the Render button
Open the output sequence with the image sequence selected and check Select File-Export-Send to Adobe Media Encoder