Realistic Immersive Content: Advanced Unreal Engine AR Template and Basic Components
When one uses their own smartphone's front camera
To film their own face
Let's learn about recognizing the face part alone
Out of the entire image
And we will implement a function that
Adds an image, such as a party mask, right on the face area
After the face recognition takes place
Moving beyond only recognizing the face as a whole
We will use functions from Google ARCore API
And track data for specific parts of face
Such as the tip of the nose or the forehead
We will be practicing this together
Today, we will use the front camera of a smartphone
And make it recognize the face
And add special objects to parts of the face
So that we can add various effects
To the face that the camera is filming
Face Recognition
Before doing that, let's first organize
All the nodes we worked with in the last class
Go to Content Drawer
HunetPlayerPawn, open its Blueprint Settings
We worked with quite a lot of nodes last time
So let's make them into functions to process them
First, this task for the screensaver at Begin Play
And these that import screen resolution
Let's put them into a function for such pre-works
Go to FUNCTIONS, and click on the plus button
And let's create a new function
Let's name it ReadyForAR
We created a function called ReadyForAR
And from this Screensaver from Event Graph
To our Game User Settings
Ctrl+X to cut this part
And let's paste it into the function, ReadyForAR
For which we use Ctrl+V
And connect the Execution Pin
This will complete our function
And let's connect this Execution Pin as well
And for this completed ReadyForAR node
Let's stick it before our 'Start AR Session'
From where it recognizes the ground at Tick
To where it spawns actor on that recognized ground
Let's create another function for this part
And name it SpawnActorToGround
Let's make that function
And now, again at Event Graph
Let's group all these nodes connected to the Event Tick
Select them all
So this is excluding Event Tick node
And it includes all other nodes connected to the Event Tick node
Ctrl+X again to cut
Make sure to go back and see if you missed anything
Here, go to SpawnActorToGround function
By double clicking, and for this body part
Let's Ctrl+V to paste what we have
And then finally connect the Execution Pin
This will complete the function
Now, back to Event Graph
Now, let's connect SpawnActorToGround
That we just made
And then compile and save
For now, Ground Detection pat is over
So we can disconnect this part
And then for 'Start AR Session'
Last time when we worked on this 'Start AR Session' part
We used a Session Config file
Which was for recognizing the ground area
And that was a particular Session Config for that
So this time, we will make another Session Config
Go to Content Drawer, and go to what we worked with last time
Which is Data Assets folder
Rick click on it
And of Miscellaneous, we will select Data assets
And then here, where we select its parent class
We will create another AR Session Config
And let's click on Select
Now, this AR Session Config
HunetARSessionConfig_Face
Let's name it like so
And then the AR Session Config file we used last time
Let's press F2 on the keyboard to add its name
Use underbar and name it Ground
By adjusting this, it will be easier to distinguish them
So now, for Hunet AR Session Config Face
In order to change its settings
Let's double click to open Details
And then for AR Settings
We don't need to change much here
But let's begin with changing its Session Type
Here at Session Type under AR Settings
This Session Type menu
For now it says 'World'
Here, 'World' means the back camera
So let's change this to the front camera
Which is 'Face' at this option
So by setting Session Type to Face
This will turn the front camera on
So it will use the front camera to recognize the face
And then for detection
All the detection parts that this has as default
We don't really need to change this
Down here, go to 'Face AR Settings'
Face AR Settings is about face recognition settings
So for Tracking Direction, the direction for face tracking
Should it be an axis value relative to the face, Face Relative
Or Face Mirrored
When we use the camera app
The camera mirrors the image automatically
I'm sure you have experience looking into a mirrored image
So if you want this to look like
The original image that the camera films
Go with 'Relative'
Or, if you rather want
A mirrored effect to the image
Use 'Mirrored'
You can choose whichever one
That you want
For me, let's just keep the default setting, Face Relative
And now, for 'Max Number of Tracked Faces'
There are cases where there are
Multiple faces existing in one image
So this is asking how many faces it should track
Unfortunately, for Google ARCore
It can only recognize one face
So whatever number you put in here
It will only track one face in result
So here, let's save our Data Assets
Let's change our AR Session
Last time we worked with the ground detection part
And today, we will work on the face detection part
In the next class, we will work on the image detection part
When our application begins
Which Session Config should be initiated?
In order for this to be decided
We will create a state here
So go to Content Drawer
At Data Assets, right click
And under Blueprint
Let's choose Enumeration
Enumeration literally means
A file where you can define enumerated constants
So you can add various states
In enumerated types
Type EDetection Type
Let's create a file called
EDetection Type
Let's now double click on it to open it
At the top, there's a button called 'Add Enumerator'
Every time you click on this
You will get a new Display Name over here
We will make three in total
So we will add three Display Names
The first one will be
Named 'Ground Detection'
Second, 'Face Detection'
Let's name it like so
And the third one, 'Image Detection'
Let's rename it like so
Now, after having created these three states
Let's save
And close the window
So after creating an Enumeration file
You can now create a function in a new file type
So here at BP Hunet Player Pawn Blueprint settings menu
Let's add a variable
Go to VARIABLES, click on the plus button
And then ECurrentDetection
Let's make a new variable named that
This function will set the current detection type
For data type, we will use our EDetection Type
Which is our Enumeration variable we created
Using this data type
This is our variable here
Let's compile, and now at Details panel to the right
We see 'Default Value'
For Default Value, we will keep its default
Which is Ground Detection
And we will make the user click on buttons at the UI
So that they can choose
One of these three detection types
So here, what we have as selected here
Using this detection type that is selected
Let's make the user choose one of these
So 'Switch'
Switch on EDetection Type, and for EDetection Type
Let's add a new Switch statement
For Ground Detection, for Face Detection, for Image Detection
Since we do not know which of these three values is selected
For these three
Each should run its own AR Session
For that, let's change the Exec Pin to this switch statement
And let's bring this part a little higher
Let's bring everything up by a little
Here, for this switch
Start AR Session, the Ground it had before
Let's connect this Ground type of Start AR Session
To this Ground Detection type
And then, same thing, Start AR Session
Create another node for the Start AR Session function
And second, let's set AR Session Config as Face type
So Session type for Face Detection
It will connect to this here
And now, to check if Start AR Session
Has changed or not
We need to check which camera is working
Between front and back camera
At a new scene
For the initial creation at widget
Let's create a new scene and a new widget
So let's go back to the main map
Let's create a scene
New Level
Same thing, Empty Level
Let's make it empty to start with
Here, 'Create'
Let's create this Empty Level with nothing in it
And let's add our Player Start to this level
And the location for Player Start is 0, 0, 0
Let's initialize it like so
And now let's save this level
Ctrl S to open this Save window
Let's save it at our Maps folder, since it is the initial map
Let's save it as Lobby Map
Lobby Map
Now at Maps folder
We have it saved as Lobby Map as seen here
Now, we will create a game mode that only applies to this map
For that, go to Blueprints folder
Right click
And add a Blueprint Class
Choose Game Mode Base
Name it as BP_LobbyGameModeBase
Let's create a file named that
At this LobbyMap, its World Settings
Let's change the Game Mode of this World Settings
If you do not see a tab at all
Called World Settings
You can go to Window up here
And you will see World Settings down here
Click on World Settings button
And you will see the World Settings window here
At World Settings tab, look at Game Mode Override
It is set as None now
Open this and BP_LobbyGameMode
Let's select BP_LobbyGameModeBase
Like so, for the rule called Game Mode Base
A class that is designated for this rule
This can be set separately for each level
For this Game Mode, when we first begin
Here, we will add to this empty space
A UI
We will have a UI image here
For that, go to Content Drawer, right click
Select New Folder
And let's create a new folder
Let's name the folder as 'UI'
For this UI folder, right click
Go to User Interface, select Widget Blueprint
And we will create a graphic UI file called Widget Blueprint
Create a Widget Blueprint file
It only has one parent class, right?
Let's now select this User Widget
So to mean Widget Blueprint, we will say WBP
_LobbyWidget
This will be the name for this widget
Open Lobby Widget settings window
First, we need a canvas to have our widget on
It's like a piece of paper to start painting
So to the left, at Palette
Let's bring in a component called Canvas Panel
Let's put this Canvas Panel to this LobbyWidget
Then, as its top route component
It will have a Canvas Panel
This green box here is
The screen to be seen
For the actual widget, like a piece of paper
Down here we have its resolution
We have its standard resolution
Nowadays, the most popularly used resolution type is
HD resolution
So its standard resolution is the HD one
Which is 1920px width and 1080px length
That's the standard resolution
In order to set for this standard resolution
Go to the bottom right of this canvas
And drag this cursor icon
And let's zoom out as much as possible
Let's fit this to the HD resolution, or 1920x1080
And once we have it
It will say 1920x1080 at the bottom here
This is the standard resolution
So the scale is also 1
Now, let's fit some things into this
First, for this widget, we will see some buttons
So drag in a button component
And paste into our cnavas panel
We need three buttons in total
That's because our choice of types
Include three different direction types
So we will have three buttons
The easiest way is to complete a button
And copy and paste that for others
So now, for this button
Let's set its anchor
Anchor means its desired location
And since we will be setting the buttons to the center
Its desired location or anchor
Its anchor should also be at the center of the canvas
Then, this pin icon
Is now moved to the center as seen here
This is now our desired point
And for this point, let's set X and Y positions
To the default, which is 0, 0
Then the top left of the button will fit on the anchor pin
But if we are to set it to the center
It is easier to have our anchor pin set on the center of the component
When we are to set its location
So go to Alignment here
And type 0.5
Which is the middle point between horizontal 0 to 1, 0.5
And also between the vertical 0 to 1
Add its middle value, 0.5
Then, like so, in the middle of its vertical and horizontal
We will have the anchor set in the center
Now, we will make the button bigger
Let's set it as around 400x100
Now, we need some letters on the button
For that, let's add Text component as
The child object of our Button component
So for this Text component
Here, its font size
Font size, around 40
Let's set it as around 40 pixels
And the content of the text, first of all
Let's call it 'Ground Type'
The text is too hard against the button
So let's add some shadow
Let's set the shadow
As around this dark grey color
Let's add around 0.6 for its Alpha value
So it has some shadow, but it's not enough
So let's set the shadow size as around 3
Around three, for all width, length and height
Let's add around 3
Now we see some depth to the text
Now, in order for the button color to pop
Let's go to Background Color to the right
And let's make it a blue button
So let's set its color like so
Click 'Okay'
When you change the button color, you must always
Have this 'Button' component selected
Before you change the color
If you change the color when 'Text' is selected
It will only change the font color
So the button name for now is
Button_Ground
Let's set it as that
Next, select this Ground Button and press
Ctrl D to make a copy
For this second button
Let's put it right below Ground Type
And for the font here
Let's say 'Face Type'
So the text is now 'Face Type'
And then, the button name
Its button name
Let's say 'Button_Face'
Select Face again
And Ctrl D to make another copy
This third button here
Let's call its name as 'Image'
And select the text inside, and its content
Instead of Face Type
Let's set it as 'Image Type'
Then you get these three buttons
Notice how they are not aligned
They are a little out of place
So we will group all these three together
So start with Ground, press Shift
And select Image, which will select all three
So here, right click
And at the bottom, 'Wrap With'
So it will be wrapped with something
Wrapped with, and at the bottom
Let's select this Vertical Box
Then, this component called Vertical Box
It will have all these three buttons
As its child objects; so this becomes a parent object
So for this Vertical Box
While it has all three buttons
There needs some space between buttons
So let's make it big enough for that
For size X
Let's set it as 400, which is same as the button's
For size Y, let's make it longer
Since we need three buttons of size 100 each
Let's start with 300
And to give 20 for each spacing
So between two buttons we give around 20
So for three, we need 60, so around 360
Let's set 360 for our height, or size Y
Now, select the three buttons
When we have all three buttons selected
Let's go to Size here and change from Auto to Fill
And on top, where it says Padding
For Padding, let's use 0 for horizontal and
Around 10 pixels for the vertical padding
Now, like so, between two buttons
We get around 10 spacing
So that's how we changed it
Like so, we have all buttons
Aligned nicely at Vertical Box
Next, in order for our Vertical Box
To move downward, or to change it position Y value
Let's bring it down by around 50
So for position Y
Let's add the value 50
And now, in order for the user to know what they are choosing
Let's add another text on top
Add a text box on this canvas
And for this text, set its anchor
At the center of the screen, and for Alignment
Let's set it as 0.5, 0.5, with the center of the screen
So when we have it back to 0, 0
Let's drag on our position Y value
Then we see our text box
That has moved upward like so
Let's keep it at around -300
And its font size, around 70
And let's align the font to the center
So its Justification, set it as center
Click here, so that it aligns to the center
After that, for this text, let's type
'Select Detection Type'
So this will be what it says, to ask for selection
Now, for font color
Let's make it yellow
So we changed the font color
Now its size should fit in the font
So it should be around 800
Or even around 100
And for height, since that of font is 70, it should be bigger
90, 100, let's say 100
Or let's even say 110
So this is what we see on the screen
It looks like a landscape view
But when we actually have it there
Since its anchor is set to the center
Even if it is vertically long now
It will be reconfigured with that scale
Let's compile and save
Now that we have the widget
Let's have its actual part
Run on the screen
So Game Mode, what we made before
Lobby Game Mode Base
Let's double click on it to run
So when it first begins, at Begin Play
We need the widget spawned there, so let's code it
At Begin Play, drag on its Execution Node
Search for Create Widget
And you will see Create Widget Node
With this Create Widget Node
Here, this WVP_LobbyWidget that we made before
WVP_LobbyWidget file
Let's set it so that it creates that
So this created instance
In order for it to show on the screen
Add to Viewport
This will move it to the screen
Let's now compile and save
Now, let's run this
Now we see the UI pop up like so
For now, since we have nothing connected to the buttons
It's now time to connect the button functions
Here, even if we press on that button
Detection Type
This Detection Type that we created before
Even if we change that type
When we move between levels or maps
For example, from Lobby map to Main map
All that data will be lost
So in order to save that data
We need a class that will keep these
So, at Blueprint folder
Right click
Create another Blueprint class
Now we need Game Instance
We need to create a Game Instance
Since it doesn't have it here
Click on ALL CLASSES here
Open the search tab and
Search for 'Game Instance'
And we have a Game Instance class
So select Game Instance class
So select it and name this new Blueprint file
As BP_ARGameInstance
For Game Instance, we will create a variable
A variable here will be
Selected Detection Type
That's the name for this new variable
Selected Detection Type
Data type, EDetectionType
That variable will be our data type
Compile
Once you compile, you'll get this default value at this Details panel
Set its default type as Ground Detection
And let's save
Now close the window
Project Settings
Edit, Project Settings
And go to Maps & Modes
At Maps & Modes, go to the very bottom
And this Game Instance class, let's change it to
What we just made, BP_APGameInstance
And then, from what we made
Let's go back to our widget
Select Ground Type button, and in order to
Connect functions to run when the component is clicked
At the very bottom, at Events, OnClicked
Click on the plus button for OnClicked
When this button is clicked, the first thing is
To bring in Game Instance
So to get that Game Instance
Get Game Instance
Add a node called Get Game Instance
And this Game Instance we currently have
Let's change it to 'Cast to BP_ARGameInstance'
Let's change that class
And then at BP_ARGameInstance
This SelectedDetectionType
Let's change this value here
To do so, we need Set
Let's connect Set Selected Detection Type to the node
Now, connect all the Execution panels
Since we just clicked on Ground button
When we click on Ground button, its Detection Type
Will be set as Ground Detection
And then, same thing
Back to Designer panel
At Deisgner
The second, Face Type button, its OnlClick
At the bottom, at OnClick event, click on the plus button
For OnClicked, this same part with the same nodes
We will Ctrl+C and Ctrl+V to copy and paste
Let's now connect the Execution Pin
And here, its Detection Type
Instead of Ground Detection, let's make it Face Detection
Finally for Image Type button
Click on the OnClick button
And for OnClick event for the button image
Same thing, copy these three here
Ctrl+C and Ctrl+V to copy and paste
And connect the Execution Pin
And then for Detection Type
This time, let's change it to Image Detection
And then compile, save
Now, when this Game Mode button is clicked
At Game Instance
Its Detection Type will be saved
And there's something to do for all these three
For all of them, we need to change their level
Let's try Open Level(By Name)
So here, the level name is
Since we're going to the main map
Let's name it MainMap
And let's copy the original name using Ctrl+C
And Ctrl+V to paste
Keep copying and pasting the nodes with Ctrl+C and Ctrl+V
So that for each Set function
Has the Execution Pin connected like so
And then compile and save
Now, this widget function is complete
Now, let's say that
We clicked on the button and went to the main map
Double click on MainMap to open it
At MainMap, something we've been working on
BP_Hunet Player Pawn
Double click and open its settings again
When it reads this Current Detection
Before it does so, DetectionType at Game Instance
Let's move it to this variable
Before ReadyForAR
Let's push it a little to the back
So before this, something we need it
Just like before, Get Game Instance
Get the current Get Instance value
And we will cast that
Cast To BP_ARGameInstance
So we will connect the Execution Pin here
Once the casting is successful
Let's push these nodes a little further
Then there is the Game Instance value
And for this Game Instance
We'll use Get for Selected Detection Type
Get Selected Detection Type
And this value, Set ECurrent Decetion
The value set in the previous UI for Game Instance
For this CurrentDetection variable
Let's copy it in there
And then the saved result
This Current Detection
You can now remove this node and connect it here instead
And the effect is
If the value that came from Game Instance is
Ground Detection
Then the top one, Start AR Session is run
And for Face Detection, the bottom one
Start AR Session will be run
Since there is no image
Because we are working on it the next class
Let's just leave it empty for now
And then same thing at this Event Tick
Based on Detection Type
What this runs will be different
First, get Current Detection variable
And from the Current Detectino value
Switch on, this Detection Type, for each
It will run different things when we connect it like so
For Spawn Actor To Ground variable that we worked on last class
It will be run for Ground Detection
Adding Image and Effect
Let's start implementing
Functions for Face Detection
Last time, when we worked on Ground Detection
Remember the function node we used?
Get All AR Geometry
Geometry is for ground detection
But we could assign different types for detection
So now, let's use this thing at the bottom
Get All AR Geometries By Class
Let's create this node
This class that we are trying to detect
Is type Face
So we will go with ARFaceGeometry
Of the values we got, this value
Or Length
Should be greater than 0
So Greater, at least one
So that means there is a face that has been detected
However, since our Max for face detection
Is one
It is impossible to detect more than one face
So either greater than 0 or is 1
They will result in the same thing
If greater than 0, it has something detected
Then, for its branch
If greater than 0
Meaning, if it detected at least one thing
If it has at least once face detected
Meaning, for True
Using that Face ARGeometry that we just got
Get, the no.0 one
The first one, since it only detects one face
The first one
No.0, for this one
Get Local To World Transform
It will get that World coordinate information
The World coordinate for the detected face
Using Get Local To WorldTransform
So at this point
Let's add something like a mask
So we need to create a mask for this
For this image, you can use the file
That I provided in advance
The mask image, use Ctrl+X to cut
And at our Unreal Project
At our HunetAR folder, Content folder
Textures folder
Ctrl+V to paste
So this is Finder for Windows
And back to our Editor
It says that we have a new image
And it asks if I want to import it
So say Import
So let's click on 'Import'
And then it says
This file has been modified to fit Unreal
So here, let's create an actor
So open Content Drawer
Right click on Blueprint Class
To create a new Blueprint class
Let's create it in type Actor
BP_MaskActor
That will be the name for this Blueprint file
Double click to open the window
Basically, we do not need anything else here
We need one Plane Mesh and one Pot Mesh
Plane, click on Add here
And add a new Plane Mesh
Static Mesh
Plane Mesh, the image is showing here now
On top of this
For that
In order to paste the image here
Go to Textures folder, the mask image we imported
Double click on the mask file to open
And its texture group is set as World
Since we are using it for UI
Let's change it to UI
And we will apply Alpha values or transparency value
And back to this window
See how Alpha value has been applied?
For mask, right click
And select Create Materials
And we get a Material file
The name will be M_Mask
Let's create a file with that name
Double click on M_Mask to open
Push its texture sample to the left
For this M_Mask
For this Material default setting node
To the left, at Details
It says 'Blend Mode'
For Blend Mode, Opaque is
Only for showing opaque image
In order to allow for transparent images
Let's change it to Translucent
So we have Translucent
And this will add Opacity here
Rather, it will activate it
Here, Alpha value for transparency
Let's connect the Alpha node to Exec Node
To Opacity
And then save
Here, the transparency value is applied
So that we only see this image
That's how we set this up
Now, let's close the settings window for Material
Let's grab our new Material here
And drag it into the Materials
And now we see the mask
But for this plane, the size is
A meter by another meter
And a mask that big isn't for a human face
So in order to change its size to something wearable
Let's change its scale values, width to around 0.2
Which is around 20cm
And length, 0.1
So 10cm
So here, width 20cm, length 20cm
That's the plane size for this mask image
And then for this, let's change it into World
For World coordinate, where is the front?
This red arrow here
That's the 'front' direction of this World
Notice how the mask is lying on the ground
So let's change it using rotation tool
To erect this mask to 90 degrees
And let's turn it by 90 degrees to this side
In order to set it up like this
So, the face that this detects
Its coordinates are now World coordinate
Which will be around the center of the face detected
But this mask isn't to be located
At the center of the face
Which will be around the tip of the nose
And this needs to be worn a bit higher than that
So in order to resettle that center
Let's bring the plane up by 5cm or 6cm
Let's bring it up by 6
Here, this decimal point isn't clean enough
So 90, 0, -90
That will keep the rotation value we had
Compile and save
Again, Hunet Player Pawn Blueprint
Back to the settings
So to that detected face
We worked up until using World coordinates
And then, for that location, what we are to do is
Spawn Actor From Class
And let's spawn an actor
And let's spawn BP_MaskActor
We will be spawning a mask actor
And then the location, we can use this
The location of the detected face
Here, Get All Geometry
The one we got as Face Geometry
The 0th one, the first detected one
Its World coordinates are brought in
And to that World coordinate
We will spawn and add the mask actor
For Collection Handling Override
To make it spawn no matter what
Let's set it as 'Always Spawn'
Compile and save
Since we are done with the setups
Now let's package it to put it up to a test
Last class, for project settings
I said that for Android
If you made a new level
You need to add it here
So go to this Packaging tab
And under Advanced
There is 'List of Maps'
And at 'List of Maps'
We have up until MainMap
But the one we just made, Lobby isn't added
So click on the plus button
And click on this dotted area
And you will see our LobbyMap
So double click on LobbyMap to add
Select LobbyMap
And at 'Maps and Modes'
The default map, the first one with the build
The first map to run
The game's default map
Now we can change it to our LobbyMap
Change it to LobbyMap
Since we are now all set
Let's work on the packaging
At Platforms, go to Android
Click on Package Project
Same as what we did last time
We mad our Package folder
So select the folder here
And let's start the packaging
When packaging is done, we need to copy the file
Go to where we have our Hunet AR Project
At Package folder
Go to Android_Multi folder
And you'll see an apk file
Ctrl+C on this apk file to copy
Ctrl+C to copy
And on our smartphone
We made a folder called Unreal App
And the file that exists here
Let's press Del to delete
And then Ctrl+V
In order to copy the file we just packaged
After copying it
Let's go to the installed files menu
Double click on Hunet AR apk file
And run its package installation program
Let's select 'install'
When the installation is over
Let's open it
Press 'accept'
And we see the detection type
Let's go for 'Face Type' in the center
Then this will give us the pop up for the front camera
Let's choose 'allow only when using the app'
And we get the front camera like so
While it does create the mask
Every time i move my face, Tick respawns it
So there are too many
Let go back to Hunet Player Pawn
And let's not spawn every Tick
When first spawned, these return values
Let's make them into variables at Promote Variables
This new variable, NewVar
Let's rename it
NewVar's name is
Mask_Instance
That's the variable name
Then here, see how the name changed to Mask_Instance
Let's set at this Mask_Instance variable
So this won't spawn every time
This time, we will get a value at Mask_Instance
And for that, Is Valid
We'll connect Is Valid function
To check first whether the created mask is already
In the variable
True button, let's connect it
To this Exec here
And if Valid
Then the mask is already spawned
So here, we'll go for Set Actor Transform
Set Actor Transform
For target, we can just use Mask Instance
Let's connect Mask Instance as the target
Like so, let's connect the target
From Mask Instance
And if Invalid, we need to spawn one
Since Invalid means that there is no value to the variable
Let's connect it so that it can spawn
At Spawn Actor, Mask Actor
Compile
I forgot to set the location
Local transformation location value
The facial location is the same
So let's connect that facial location value
Compile and save
And let's try packaging again
Same thing, choose Face Type
Allow only when using the app
So while allowing the front camera
We raised it up by 6cm before
And i guess 6cm is a little too high
Anyhow, whenever I move my face
We can see that the mask respawns
With the new location for the face
But the problem is, if i turn my face sideways
It does not recognize the rotation value
So since it is set at the center of the face
It follows the face well
But the rotation value, when the face
When the face location is fixed
When I just turn my head around
Notice how it fails to follow promptly
So let's get an actual location on the face
And set this again with that point
Detecting Particular Facial Point
Before that, as we saw
The position was a little too high on the face
So let's move the mask actor plane location
A little less than 6cm
Let's say around 4cm
Save
And let's not make it the center of the face
Let's instead position it on the tip of the nose
So let's go back to Hunet Player Pawn settings window
To set the location
This time, at Geometry Class
We can bring in the World coordinate right away
So this center coordinate for the face
Let's disconnect this pin
Let's actually bring it down a bit
And here, this first value
Let's cast that value
Cast to GoogleARCoreAugmentedFace
Let's change it to this class
A pretty long name, right?
It's very long, so let's put it here
Let's not connect the Exec Pin right away here
Only cast when it is True at this branch
Let's change the Exec Pin connection like so
And then at GoogleARCoreAugmentedFace
At this actual face
Get local to world transform of region
Region meaning a specific location
Such as eyes, nose or the mouth
This will bring World coordinate for those
Get Local to World Transform of Region
Let's add this node like so
See, when we set a particular region
We can go for three parts
The Nose Tip
Which is the tip of the nose
And then the forehead
The left and the right of the forehead
And when it says Forehead, it means
Anything higher than the brow
That's what this calls Forehead
Let's actually use that region
Left and Right of Forehead
Then it will specify area above the brow
So for us, we can use these three
To calculate the specific regions on the face
As long as we know where the eyes, nose and mouth are
We will use the tip of the nose
As where the mask actor will be spawned
So let's push these nodes a little further
And let's connect it to Valid here
Let's bring this up a little
And let's connect it to our Valid Exec Pin
Others remain the same
But while this used to connect directly
Directly connected like so
And it began with the face center coordinate
But this time we use Nose Tip
So we bring in the nose tip from the face separately
So when this location is created separately
Or if already separated, when the location is changed
This will connect like so
Compile and save
Let's try the build again like so
This will now detect the face
And since it uses Nose Tip location
When we turn the face
The image will rotate with the face
As we see here
So compared to before
When we used the face center
When we use Nose Tip
This becomes more accurate
So that's how
We use face detection
Great job everyone for powering through such a long class
Next time, we will use the marker image and
Implement the Image Detection part of this
Thank you
Let's summarize what we learned today
Fist, we created Enumeration file
And allowed for distinguishing Detection type
Based on the user preference
Using a widget that we created
Second, we worked with AR Session Config Data
And set the session type properties
To allow for face detection function
And to allow for the actual use
Third, the facial region
Such as Forehead or Nose Tip
Using these particular detection regions
We were able to display our actor
That syncs well even when the face moves around
It's not only for actors
We can also add things like Niagara effects
To add various effects
Face Detection Data Asset Preparation In DataAssets folder, create HunetARSessionConfig_Face file for face detection
Open settings, set Session Type as Face Enum Class Data Creation Create Enum class file
Double click on EDetection Type file, +Add Click on Enumeration to add three Enum type constants
At BP_PlayerPawn settings, create a variable ECurrentDetection with EDetection Type as data type
At Event BeginPlay node, use Switch node to branch AR Session nodes to distinguish detection types
Data for Start AR Session node connected to Face Detection Exec Pin: set HunetARSessionConfig_Face file
Setting LobbyMap Select Empty Level, place PlayerStart actor at default World (0,0,0) Create a new Blueprint file at Blueprints folder
Parent class: GameModeBase class Set the file at GameMode Override at World Setting tab
Create Lobby Widget Content Drawer, create a folder called UI
+Add - User interface - Widget Blueprint to create a new widget
Parent class: UserWidget Open widget settings and add Canvas Panel at the left hierarchy panel
Create three Button components as child component of Canvas Panel
Face Detection Create AR Game Instance Create a new Blueprint file, set parent class as GameInstance
Open BR_ARGameInstance file settings and create a variable with EDetection type as data type
At Game Instance Class, set BR_ARGameInstance
Click on a button at Lobby Widget to implement the function Open settings and select Ground Type
Click on + at OnClicked event Add a node to get BP_ARGameInstance
At BP_ARGameInstance class, set Selected Detection Type variable to GroundDetection
Connect Open Level node and type MainMap to the level to move to
For Face Type and Image Type buttons, also set Detection Type for the designated OnClicked events
Adding Image and Effects Creating Mask Actor Right click on Mask texture file, choose Create Material to create a new material
Blend Mode: Translucent Connect A Exec Pin of Texture Sample to Opacity at M_Mask Node
Create a new Actor Blueprint file called BP_MaskActor Open blueprint settings and add Plane component at Components panel
At the right Details panel, set M_Mask file under Materials
Face Detection Implementation Open BP_HunetPlayerPawn settings, connect Switch node for EDetection Type at Event Tick node
At Face Detection Exec Pin, connect Get All AR Geometries By Class node for face detection
Set Geometry Class to detect as ARFace Geometry
-If Branch node is True, connect to use SpawnActor node to create BP_MaskActor
To avoid overlapping spawn, create Mask Instance variable and set Reference of spawned actor
Location Detection for Facial Region Getting location for facial region GoogleAR Plugin supports Left and Right Forehead and Nose Tip location values
Connect Cast To GoogleARCoreAugmentedFace Node to ARFace Geometry Node
To get coordinate for Nose Tip, set Nose Tip for Region at Get Local to World Transform Of Region