Let's practice how to use rear camera
on smartphone to recognize floor of real world
In order to make camera to recognize floor
use function to prevent standby mode set on your phone
while it works
and in order to find the center of screen
we will implements function to get resolution size of each phone screen
And not only just recognize the floor
but create specific object we prepare
depends of area or slope of floor recognized
recognition of floor on real world
start Unreal Engine
Now it's not for sample project
in order to work real projcet
let's create new Level
Go to File menu on the left top corner and select New Level
to make new Level So select
Empty Level
and click Create
to create new Level
Since AR contents is basically
creating 3D objects on real world where camera is capturing
In order to fill the backgrounds
with the picture camera captures
we selected Empty Level
To develop AR contents
let's do some basic necessary works
First, we need to set player start point
where player will be placed
Therefore on Place Actors
drag Player Start
and place it on the screen
Location at 0, 0, 0
and Rotation at 0, 0, 0 as well
In order to make objects visible
when some objects are placed on screen
we need lights
therefore let's place Directional Light
which gives the effect of sun on the map
It doesn't matter where you place it
Since Directional Light
gives the effect of sunlight
wherever you place it, place here or place here
since the sun's light goes everywhere
in the same direction on screen
it really doesn't matter where you place it
But the direction of light is
the white arrow you can see here
Source Angle on here
it can be changed by manipulating Source Angle
That's all you should know
Let's just keep the default value for now
and place it little above from 0, 0, 0
Then we need to save Level
So go to Contents Drawer, on Content folder
Click Add button and New Folder on top
Select New Folder
to create a new folder
Name it Maps
and we will save current Level
In Maps folder
name it MainMap
and save it
Then in Maps folder
you can see the file
named MainMap is saved
Next, we will create Player Pawn which will appear
on Player Start position
So in order to make Player Pawn
in Content folder, go to Add then New Folder
to make another new folder
and name it Blueprints
And go into Blueprints
right click on empty space
and select
Blueprint Class above
Pick Pawn class for parent class
Select Pawn class
Name new Pawn BP_HunetPlayer
name it like this
Open settings of BP_HunetPlayer Pawn
we only need camera on HunetPlayer
Click Add button on Components
camera, on search field
type camera
You see Camera here
Let's add
Camera component
Then on viewport window
you can see camera mark like this
Compile and save it
Back to MainMap
Player Pawn we just made
in order to make this HunetPlayer Pawn to default Pawn
we will make a game mode
Right click again
select Blueprint Class
for parent class
select Game Mode Base
For the name of blueprint file we created
BP_ARGameModeBase
set it like this
Double click ARGameModeBase
to open settings window
Default Pawn Class on ARGameModeBase
Default Pawn Class on the right
to BP_HunetPlayer we just created
set it to BP_HunetPlayer
Compile and save it
In order to make this ARGameModeBase we just made
to main game mode
Edit menu on top, click Project Settings
to open project settings window
Click Maps & Modes tab
on the left
Set Default GameMode
to BP_ARGameModeBase
change it to this
If you open Selected GameMode here
You can see Default Pawn Class is set to BP_HunetPlayer
And in case of Default Maps as well
set Editor Startup Map
to MainMap we just created
Game Default Map which is executed first
when packaged to apk file
set this to MainMap as well
Change it to MainMap like this
Come back to MainMap
Now the next thing we are going to do is
double click HunetPlayer to open it
On BeginPlay
in order to use AR at first
we need to start something called AR Session
AR Session is
simply speaking
where this apk file will be executed?
On the phone
It is intermediate equipment
to synchronize lifecycles
of Unreal Engine
and camera on your phone
So pull out from BeginPlay node
type Start AR Session on search field
search it
Here is Session Config
About session which will be opened
it meant to be put file about session settings in
But we haven't made
Session Config yet
so we have to make it now
On Content Drawer
we will make folder again
On Content folder, click Add
Then New Folder
to make another new folder
And the name of folder
name it DataAssets
Inside of DataAssets
right click
and select Miscellaneous
find and select Data Asset
among these list on the right
In the case of Data Asset, we can select
which class this Data Asset will be
Among them ARSession Config
to set about Session
select ARSession Config file
click Select
to create Session Config file
Name of Session Config file we made
is HunetARSessionConfig
name it like this
Double click HunetARSessionConfig to open
Then you will see a variety of setting values
Among the setting values
first thing we are going to do
is recognize the floor
Under this AR Settings
First, there is Session Type
Session Type
Session Type is set to World
There are options such as World, Face, Image
recognizing floor means
recognizing World
So we will set this to World
Especially in this case of World
there are front camera
and rear camera on your phone
It's to turn rear camera on
Set it to World to select
turning on the rear camera as type
Above that there is World Alignment
align method of World
In order to make it align with the direction of gravity
select Gravity
And among the detection types
Horizontal Plane, Vertical Plane
We can detect both of them
Both of them is checked as default
Horizontal Plane is
Horizontal therefore detects horizontal surface
such as upper side of desk or box
Vertical detects vertical surface
such as wall or window
So we can detect
both Horizontal and Vertical
If we detect too many surfaces it would be dizzy and confuse
so remove Vertical
and leave Horizontal on
to detect only floor, horizontal surface
To adjust the focus of the camera automatically
we have to turn on Enable Auto Focus
Light Estimation Mode
If the picture camera is capturing is too bright
then reduce the brightness of camera
if it is too dark, increase the brightness
Light Estimation Mode working as aperture
set to Ambient Light Estimate
to measure occlusion light
To measure how dark surrounding is
with Ambient Light
There are various things
but we don't need to touch anything else
so leave them as they are
and adjust only these part
Seession Type, Horizontal
Vertical and Detection
check only these
Click save to save it
and close SessionConfig window
Now that we made session config
Come back to BP_HunetPlayer
Start AR Session node
we made earlier
on Session Config
put HunetARSessionConfig file
we just made
Now on Session
Session is executed now
We are keep going to move camera
we need to know which plane of objects
was recognized every frame
And also need to check
whether things are recognized well
So drag execution pin from Tick node
and type get all AR geometries on search field
and select it
to get every target
AR camera recognized in the form of array
let me draw every array
that camera recognized on screen
Pick execution pin and drag
type debug draw
if you search it
there will be node
called Debug Draw Tracked Geometry
Debug Draw Tracked Geometry
This node can draw
selected Geometries on screen
But it detects as array
and indicate one by one
therefore use loop statement
to make it indicate multiple things at once
So drag this node
and execute For Each Loop
Use For Each Loop
connect execution pin
To make it Debug Draw from beginning to the end
which means it will draw
each element values in array
Draw lines
Probably it will draw the outline of plane we detected
If you open it by clicking arrow below
we can set things such as
the color and thickness of outline
Change settings
to draw green outline
of target we detected
Then the thickness
5 pixel would be too thick
It'd be enough to set it 1 pixel
let's just change it to 1 pixel
This is for how long this outline will be remained
We are doing this on Tick
Therefore it doesn't need to be remained
Since we will draw again every Tick
compile and save it
Then back to the map
package it once
Package it
Same as before
select Packages folder
click Select to start packaging
Once it is done, as same as last time
under Unreal Project, under HunetAR
Where our projects is located, under Packages folder
it's the apk file under the Android_Multi folder
Ctrl + C to copy apk file
We made folder named UnrealApp
last time on the phoe
To make it easy to be found
exist file in this folder
press Delete key to delete it
and Ctrl + V
to paste what we packaged
When it's done, unlock your phone
APK on your phone
apk install file on My Files
double click
HunetAR we just pasted
and install it with install program
Since we installed once last time
so it will be overwritten
So when you overwrite it
not asking if you would install it
it will ask if you would update
Then tap update
to overwrite
It is finished now
then tap open
Now then
if you try to aim surface
you will see the surfaces are starting to be recognized little by little
Planes are recognized little by little
Now it's narrow space
and have a lot of bumpy surfaces
so the recognition might be take some times
One of the reason it takes time to recognize is
that there are dark
and shadow surfaces here and there
but if you keep aiming slowly
then you would see the recognized surfaces are increasing gradually
You can see the outlines are increasing
Now we can see the surfaces
are recognized like this
Now it's confirmed
that camera is recognize plane surfaces well
Then the part where we debug
was only for checking
so disconnect it
and move these nodes aside above
We are not going to process collision for now
so let me put aside below where we it can not be seen
move Tick little below
For this
note that these are nodes indicating outlines of surface recognized
let's mark it like this
We can connect these node
whenever we need for checking
Prevent standby mode and function to get resolution information
For convenience
we will add some nodes
First, when we are aiming
AR camera to recognize plane
While we are doing it
screen could be turned off
by working of screen saver or standby mode
Therefore, in order to prevent it
we will add a node to turn screen saver off
So it needs to be done only once
on BeginPlay, on Start AR Session
drag execution pin out
And search control screensaver
Then you will see Control Screensaver node
On Control Screensaver
uncheck Allow Screensaver
to set for screensaver not to work
And now all geometries
it will detect geometries we recognized
Now with geometries, we recognize the center of your phone
when you aim center
if there is plane on the center
of screen we are aiming with camera
then we will display a object
So let's make a object
we'd like to generate there as blueprint
There is a image file
It's Indicator.png file
Ctrl + X to cut this image file
cut it and
under Unreal Project, under HunetAR project folder
where our projects are located
and make a new folder inside of Content folder
New and Folders
So let's make a folder
named Textures here
Textures
And inside of this Textures folder
Ctrl + V
to paste Indicator image file
Now since new image is added
you will be asked
if you would import on editor
then import it
Click Import button
Then if you look at Content Drawer
we made Textures folder
So inside of Textures folder, you can see
there is file named Indicator
But texture file is just white
It's hard to see but
it is a circle shape
with some pointy arrows on the transparent background
This image supposed to be shown like this
but it isn't now
Double click texture
to make texture image sticks out
on the right side here
change Texture Group
from World to UI
Then save it
and close settings window
and go back to Textures folder on Contents Drawer
now you can see
the transparent and opaque part
of Indicator file like this
Then based on this texture
we will make a Material
Select texture and right click
Then Create Material
Cilck Create Material
and once new Material file is created
name it M_Indicator
name it like this
Then double click Indicator Material file
to open settings window
Now the Texture Sample is just white like this
but the real texture is, if you select this node
you see Indicator is assigned to Texture
But the transparent value is not applying
Here is M_Indicator, the basic node
Select this main node
On Details pannel on the left
there is Blend Mode here
Opaque is used
to indicate opaque image
But for the image
which has transparent value, alpha value
you need to select Translucent
So change it to Translucent
Then enabled sockets on main node would be changed
And Opacity would be enabled
This alpha value, this is transparent value
connect this transparent value to the Opacity
Now the transparent part
and opaque part has been split like this
We will indicate it
on plane, flat surface
on preview viewport, right in the middle among the buttons
let's click plain button
Then you would see image texture
on flat square surface
Now we can see it better
But it still can not be seen well since it is white
It could not be seen well when it is on camera
so let's add some colors to it
In order to put some colors
right click on the empty space
and type VectorParameter
VectorParameter node will be searched
Select VectorParameter node
and name it Color
Put Color node little above
drag this RGB socket and type *
then Multiply node will be searched
It's to multiply
Socket A of Multiply node is connected
And Indicator texture we chose earlier
connect RGB value of Texture node
to B socket of Multiply
And the result of multiply
will be passed to Base Color
Then which color are we going to add
in order to make it stick out
let's make it yellow
Yellow color and set alpha value to 1
make it opaque then click OK
So let them multiply with each other
How it would be like?
You will see
Indicator image with color like this
Click Save to save Material settings
If it is saved, close Material settings window
At last, go to Content Drawer
go into Blueprints folder
let's put Indicator Texture we made
to actual 3D modeling
to square plane modeling
Right click
and create Blueprint Class
I'll make it as Actor since it needs to be placed on World
blueprint inherited Actor class
name It BP_Indicator
Double click BP_Indicator file
to open settings window
Under the DefaultSceneRoot of Indicator
click Add button
Select Plane here
Plane Mesh
Set a Plane Mesh like this
If you compile it once
on Plane Mesh, below Static Mesh
there will be Materials
In search field on this Materials
type M_Indicator
then you will see Indicator Material
M_Indicator Material
Assign this Material
we created before
Then on Plane Mesh
you will see Indicator is indicating like this
But let me change it to World coordinate
with globe mark
On World coordinate
which way is the front?
The red arrow
This red arrow refers to forward vector
blue is up vector
and green is right vector
And the direction of world is red arrow
In order to match with forward direction
this needs to be rotated
So switch it to rotation tool
rotate it 90 degree
Therefore when switch it back to move tool
front direction
and the arrow part
on the front of this texture
should be pointed same direction
Since decimal point is irritate
so let's just type
90 manually
So that rotate it 90 degree
Then this Plain Mesh has
1m of width and depth
But this 1m-long Mesh
is too big to fit in
when we are aiming in short distance
It is too big for small pace
so let's make Indicator smaller
In order to make Indicator smaller
Set Scale to 0.1, 0.1, 0.1
to make
Indicator
10cm long
Now it is very small
we need to get real close to see it
anyway, 10cm long, by setting to 0.1, 0.1, 0.1
we made 10cm-long
Indicator blueprint file
Compile and save it
and close this settings window
After close the window
HunetPlayer
let me change the name
from HunetPlayer
to HunetPlayerPawn
Otherwise it could be confusing later
whether this is Character or Pawn
so I'll just name it
as Pawn
So open HunetPlayerPawn settings window
Recognize floor on Event Tick first
and if there is recognized floor on the center of screen
then we will indicate
that Indictor on the floor which is recognized
Therefore we need the value of the center of screen
but resolutions are different
depends on each phone
so the center of resolution of our phone
we are using right now
it would be half of width of resolution, half pixel of resolution
and half of height pixel
to be the center of screen
But we don't know what pixel
width and height are now
so we will get those values
In order to get pixel values of phone
right click
Type get user settings
Game User Settings
and add
GetGameUserSettings node
Now we will get the resolution
So here at Return Value
type get resolution
It is resolution
Search get screen resolution
then you will see there is
Get Screen Resolution node
You can get resolution values of screen
And it would be consisted of X value and Y value
We will save these values
Drag this
type promote to variable to save it as variable
and set it
like this
The name of variable is set to NewVar
Let's change it on Details panel
On Details panel, Variable Name here
name it ScreenRes
for Screen Resolution
So the size of screen
And then compile and save it
Then it is saved
in the name of ScreenRes now
On this Get All GR Geometries
if you search Line Trace
You would find Line Trace Tracked Objects
Line Trace Tracked Objects
Line Trace Tracked Objects detects
with Line Trace
what I'm tracing right now
So use Line Trace Tracked Objects
from specific location of screen
shoot Line Trace if there are
geometries recognized there
Unlike Line Trace shooting from World
we shoot it from camera
so it is a bit different function
in terms of shooting Line Trace
by analyzing image of real world
So the ScreenRes
that we saved earlier
Get this Screen Res value
and pass it into here
but if we just pass it
then the resolution itself will be passed
so it would be right bottom corner
Half and half
In order to make it half of width and height
Use split to separate
And multiply each value
So Multiply node
First, change this to Float
and set it to 0.5
Multiply 0.5
And connect to another Multiply node
and change this to Float Single
and set 0.5 to it
Therefore split the half values of width and height of resolution
and connect to each Screen Coord
X to X, Y to Y
pass them in this way
And Line Trace Tracked Objects
there may or may not be it
Center of screen
may or may not point at plane
Therefore we need to check
whether there is object we are pointing at plane
So the length of this array is
greater than 0
Then if it is greater than 0, then there are one or more
which means something is detected
If this value is True
on Branch node, if greater than 0 is True
it means there are one or more plane is recognized
So if there are one or more recognized plane
get value of index 0 from array
using Get node as reference
The first one of array, most of the time
there won't be more than two recognized planes
So get first recognized plane
And get one location value from it
Type get local to world transform
to convert local coordinate
of recognized plane to world coordinate
So convert coordinate of plane we recognized
to world coordinate
and in accordance with it's location and rotation values
we will generate Indicator
Therefore if it is True
type spawn actor from class
to add
Spawn Actor node
Which class would that be?
BP_Indicator
BP_Indicator we created earlier
we will generate this
BP_Indicator
In terms of Spawn Transform
right click and select Split struct pin
Location, Rotation, Scale
then it splits into these three
And this value, plain we recognized
if we split this
it will be split into
Location, Rotation and Scale as well
The reason I split it is
to match location in accordance with recognized plane
and match rotation in accordance with direction of rotation
since the flat surface could be slope a bit
so match the direction of rotation and all
but I won't reflect Scale
Because if the recognized target
do Scale as well
then the value of Scale we set before
could be ruined
In terms of Transform Scale
we will keep it as 1
So no matter how far
or close it reflects
if you want its size to be even
you can split it like this
And if you want the size of Indicator decrease when it is far
and increase when it is close
if you want it to be in accordance with the distance
then you can connect Scale
to here like this
But now, I'll force the scale to 1
to make the size of Indicator
is always even for your practice
To make it able to be spawned unconditionally when it spawn
set it to Always Spawn
Now compile and save it
and let's test it
Before that, let me make something important clear
We are getting the values of resolution
But these values
we can not get these values as soon as app starts
Because even if app has been started
it takes physical times
to read values of machine
and little bit slow for phone
give some delay value before here
to give machine enough time
to spend and read values
and get Game User Settings
And get values of resolution
So I will give Duration here
It is set to 0.2 as default
and yes, I think 0.2 seconds would be enough
Delay for 0.2 seconds
then read data
Read resolution data of your phone
So compile and save it
and package once more
For the test
still in the same folder, in Packages folder
let's package it here
Once packaging is done, then
let's install on your phone again
Copy apk file from Packages folder
and delete exist file
in your phone and paste this
Now then unlock your phone
and install file in My Files
update this
once again
And start it when it's finished
Now then you would be able to see
Indicator at the center of screen
It is recognized well enough
but problem is that Indicator
is remaining and stacking
It is generated every Tick and it covers the screen gradually
Generate object in accordance with slope of floor
So come back to HunetPlayerPawn
and let's edit little here
Whenever it is recognized
and there in recognized plane every Tick
it should not generate unconditionally
let's make this variable
make generated Indicator to variable first
The name of this variable
I will name it IndicatorRef
Ref for Reference
make a variable
named IndicatorRef
Compile it
Once it spawn then save it
to variable IndicatorRef, so at first
shoot from Line Trace of the center of screen
and if it hit to recognized plane
rather than generating it unconditionally
but it will be checked once more
Move this below like this
If this is True
the IndicatorRef
we will check this one more time
To see whether there actually is value in it
add one is Valid node
and connect True here
So that whether there is something assigned to Indicator variable or not
If not we needs to generate it
And assign it to variable
That's what we will do when there is not
If there is generated one already
then, on IndicatorRef
use Set Actor Transform
to change Transform
And update location and rotation
Split Transform
And move these to organize
So pass this Location value
and Rotation value as well
Let me organize lines since it's too messy
set things like this
and set new value
to Location and Rotation
Therefore if Indicator already exist, then change location
and if not, then generate Indicator
and assign it to variable
By doing like this can prevent for Indicator
to be generated every Tick
Compile and save it
and package one more time
to check it
Copy this
And reinstall on your phone
Update it
And start when it's finished to be installed
Now then how does it look like when it recognize plane
In accordance with plane
you can see it is recognized
as it indicate that here center is flat surface
If you point at somewhere not plane
then it rotates while trying to find plane
Try to rotate in accordance with surface
It doesn't recognize wall yet
since it doesn't recognize wall
Indicator does not stick out well
But you can see here is recognized well
So now we recognize plane
and indicate object on that recognized plane
these are what we have done
From now on, change a variety of
objects other than Indicator
and edit blueprint file
to try to install various objects
Let's put together what we learned in this lesson
First, in order to use AR function
we practiced to create new Level
and the process to create GameModeBase class
and PlayerPawn class
Second, used Line Trace Tracked Objects node
which is specifically designed for AR
to fire Line Trace on specific pixel location
of phone screen
and implemented function
detects if it is flat surface where Line Trace was hit
Last, created Indicator Actor indicates location
and direction of recognized plane
and created Actor in accordance with floor which is actually recognized
through phone camera
Recognition of floor on real world create Level and Pawn Class for AR
AR Session which synchronize with lifecycles of Unreal Engine and camera needs to be set for AR
Set Session Type to World for plane recognition Set World Alignment to Gravity to align the direction of gravity
Uncheck Vertical Plane Detection to make it recognize only for horizontal flat surface
Function to prevent standby mode and get resolution information Prevent standby mode
Drag execution pin from Start AR Session to Control Screensaver and uncheck Allow Screen Saver
Function to get resolution information Add Game User Settings node to get center pixel coordinate of screen
Half value of X, Y of variable ScreenRes to each socket of Screen Coord Check with Branch node whether there is recognized plane or not
Generate object in accordance with slope of floor Create Actor on the location of actually recognized plane need to make generated Indicator to variable iwhen Indicators keep generated on plane
Connect IndicatorRef to SpawnActor BP Indicator Add is Valid node to check whether it has actual value or not
If value exist, use Set Actor Transform to update Location and Rotation of IndicatorRef