AR Candidate Image is
An image database dedicated for AR
We will try adding AR Candidate Image as well as
Implementing real-life image detection through phone camera
Also we will make 3D object on top of the image
That corresponds to the detected image
Also, adding onto single images
When various images in the image database
Are all shown on one screen
We will also make objects spawn which work for
Each individual image
AR Candidate Image and Real-Life Image Detection
Hello
Today, continuing from the last class
We will work on marker detection
Here, what is a marker?
When we add images for detection
And add these candidate images to the image database
When these images on that image database
Are filmed with a camera
The 3D objects that correspond to that detected image
Will be displayed on top of the image
This is what we call Marker Recognition/Detection Method
So, before we practice on how it works
All the works we did last class
Since there are many nodes for that
Let's re-organize them into functions
First, go to Content Drawer
Double click on Player Pawn Blueprint
Open its settings window
On its Event Graph Node
When Face Detection is selected out of Switch Detection Type
The part after that
Select all the nodes after that
If you missed anything, you can press Ctrl and click
To add to or remove from the group
If you selected everything
Ctrl+X to cut
And to the left, at Functions tab
Click on the plus button to make a function
Let's call this function Spawn Face Actor
This new function is called Spawn Face Actor
At the beginning of Spawn Face Actor, Ctrl+V
To paste all the nodes
That we cut before
And its Exec Pin, connect it to
Get All AR Geometries By Class node
Then this function will run at once
Which will add the mask image on top of the face
Last time, we worked on putting the image
On Nose Tip or the tip of the nose
That's how you create the function
If you are done, compile first
Back to the Event Graph
Get Spawn Face Actor Function Node
And connect Face Detection Type
That's how we can make things more succinct
And categorize everything
We worked on this part as well last class
Let's keep it as it is for now
For Begin Play part, we don't have
Much to change as of now
So, let's create an AR Session
To create an AR Session
Go to Data Assets folder
And we will create another AR Session Config like last time
Right click
Go to Miscellaneous
And select Data Asset
And same thing as before, AR Session Config
Select AR Session Config
Name is, Hunet AR Session Config_Image
Let's call it that
So this the Session Config for image detection
Double click to open its Details
This time, we need the back camera
So Session Type, World
It will be filming the World
I mentioned that the back camera is what films the World
So that's the back camera
On top, at AR Settings, at Image Tracking
At Image Tracking tab
It says Candidate Images
And we are to add some array here
So this is the candidate image
We need to add the candidate images here
But since we don't have any image now
Let's start with creating the image
We have the images Cat and SCar
Of course, you can use
Any image you'd like
But for now, let me demonstrate
Using the images I have
Using Cat and SCar images
Let's Ctrl+X to cut
When we worked on Project Hunet AR Project
We made a folder at Content called Textures
At this Textures folder
Let's Ctrl+V to paste
It says 'import' here
So click on 'Import'
And that will import the images
Let's close this section for now
And if you go to Textures folder
We see the images we had
Cat and SCar
The background is filled in as white
Block white for now
And the image we want is in there
This image size is 1024x1024
Same for this as well
1024x1024, so this is a square
These images are simple image files
Let's add these simple images files as
Candidate images for AR detection
Go to Data Assets folder
Right click
At Miscellaneous, click on Data Assets once more
We will create the candidate image files now
Here, 'AR Candidate Image'
Click on 'AR Candidate Image'
And click on Select
For this first image
Let's name it AR Image_Cat
AR Image Cat
Double click to open
Candidate Texture
It says Candidate Texture
So it says add candidate texture
Here, we will be adding
The image we imported, Cat, as the candidate image
And then, Friendly Name
Meaning, we can add some sort of a nickname
To this image
I will call it Hunet Cat
That will be its nickname
And it says Width and Height
They literally mean width
And height of the image
When this image is seen in real life
It could be a printed image
Or an image on a monitor screen
Whatever the means, when this is displayed on an actual screen
This will be its actual size
If we are to make it into a poster, for example
Or into a sticker
Sticker, poster, or on a monitor screen
Then, the actual image size
Is to be put in here in centimeter
So that's how it detects this image
In regards to that size
It doesn't have to be a perfect measurement
It just needs to be around the right size
For me, let's say 20cm
So 20cm width, 20cm height
In fact, once you add the width value
Its height will be automatically scaled
Based on the resolution
So now that we have the image
Let's save
And close the window
Let's come back
Go to Data Assets
And something we stopped making
Hunet AR Session Config, let's open its image file
And on top, Candidate Image
Let's add an array of the candidate image
So there, that image
AR Image of Cat
Our candidate image file
Let's add it here
Below says Max Num Simultaneous Images Tracked
Which is the maximum number of
Images to be detected simultaneoulsy
I guess five is a good number
To start with here
Let's save here
And close the window
So, at Hunet Player Pawn's Event Graph
When it first begins
At BeginPlay
If Detection Type is Image Detecton
It starts AR Session
And of Start AR Session
Our session here is
The Image Session that we just made
Hunet AR Session Config, its image session
That's what it runs here
And then now at Tick
It's now set as Image Detection Type
So the value of the variable ECurrent Detection
If that is set as Image Detection
Here, first of all
Let's try detecting as image
So as what we've been working on
Get All AR Geometrics By Class Node
Let's create that
At Get All AR Geometrics By Class
It will detect in the class type of
For us, ARTracked Image
ARTracked Image
This will detect this image like so
Here, the returning value is
The detected images
The array of these detected images
So here, for our candidate
Since we only have one candidate image
It will end up detecting one image
So here, at this array
Get, its 0th one
It will get the first image it has
And there, in order to check
If the image detection was successful
Let's have it color in the outline
Of the detected image
Search for 'Debug'
Debug Draw Tracked Geometry
Meaning, it will draw an outline of that detected image
It will draw the outline
To check if the detection is working
Let's compile and save
Let's try packaging now
Package Project, and package it
At Packages Folder
Let's make a package file there
While it is packaging
We can open that image
That cat image in advance
So, open this image
And since the packaging is now done
Go to Packages folder
And to that packaged file
Ctrl+C to copy
And then on my phone
At Unreal App folder, let's delete
The existing file by pressing Del
Ctrl+V
And let's copy and paste in
The newly packaged file here
Once it is complete
Let's install the apk file
When it's done, open it
Let's press 'Open'
For Detection Type, let's go with Image Type
Then, on the camera, this cat image
It is detecting the image well
And with that image, we see
The red outline around the cat image
When we move the image here
We can see how the outline is moving as well
Spawn 3D Object That Corresponds to Detected Image
So now
Let's add to that detected image
So that on top of that image
We spawn a 3D object
So we need to create a 3D object for that
Contents folder, Add New Folder
Click on New Folder
To create a new folder
Let's name it Mesh
That's our new folder
At Mesh folder, let's add
Cat_Idlf.fbx
And Texture.png
Let's Ctrl+X to cut the files
And at Project, Content, Mesh
At Mesh folder
Ctrl+V to move it there
It now says import
So click on Import
This is what we got as Cat_Idle
This fbx file is a file that has
Both its modeling and animation
So it says select is skeleton
Since we do not have
A separate skeleton for it
Let's just set it like so
And import
The warning message pops up, which you can ignore
Close the window
Back to Mesh folder
Skeletal Mesh file
And Physics Asset file
We can see that we have the Skeleton file here
But this Material file
This material added to the modeling
For now, it's just white
So something we moved along with this
Let's add Texture, which is
The image to map on this modeling
It says Material or Ult
Double click on the material file to open
It says Param here
Delete this node
And bring in our Texture file
Drag it in
And connect its RGB values to Base Color
So select Skeletal Mesh
And click on this brick icon
And we can see a preview of
How it looks like on the modeling
Once it is saved
Close the Material Settings window
And using this Skeletal Mesh
Let's create a Blueprint actor
Go to Blueprints folder
Right click
Select Blueprint Class
Since we are to place it at World
Let's call it Actor
BP_CatActor
Double click on our cat actor
Open its Blueprint settings window
Simply click on Add
And add Skeletal Mesh Component
That's all it takes
At Skeletal Mesh
At what we just added, Skeletal Mesh Component
Look to the right, at Details panel
We are to add its Skeletal Mesh Asset
Here, let's choose Cat_Idle
Let's add that Skeletal Mesh here
Since it is large now
Let's resize it
The image is only 20x20cm
And if the cat is this big
The screen won't capture this much
Let's make it as big as the image
So let's reduce it to 0.2, 0.2, 0.2
Compile and save
See how I set it as World
It's not Local but World, looking like a globe
In that case, where is the front here?
This red direction is the front
And then, notice how this cat modeling
Isn't facing the front
So let's rotate it so that it is facing front
Select Rotation tool
And rotate it by around 90 degrees this side
This value could be around -90
The Z axis
Compile and save
For now, let's make the front direction
What the modeling is facing towards
Let's save and close the window
Now, back to Hunet Player Pawn
Remember that outlining node?
Let's move this outlining node aside for now
We will be only using it
When we need it for debugging
So this can move aside for now
For now, let's check first on the detected image
Whether that image really is the image we added
As the candidate image
So, Get Detected Image
Search for that
And we see Get Detected Image Node
Get Detected Image
Let's get that AR Tracked Image Type
This image, Get Friendly Name
So what is Friendly Name here?
We have that image
So for that candidate image, that cat image we had
I called it Hunet Cat, remember?
Hunet Cat
Hunet Cat, so with that Hunet Cat image
On top of that, we can add that modeling
That corresponds to that image
So for that, let's create a variable in advance
To the left, at VARIABLES tab
Click on the plus button to add a variable
So let's pair the image and the modeling
Meaning, what modeling will this image create?
That will be paired in this
Data type that we will create now
So for the first image, its data type
Its basic data type is string
And to the right, at Details panel
It's not just a single image, string
It's Map, which is a special data type that
Saves data in key and value
It's also called dictional
So for data type
Let's change it as Map
The data for Key is String
And data for Value is
Let's say Actor
Actor's Class Reference
Since we are assigning a Blueprint file
It is assigned as Class Reference, or the reference type
Of the actor class
So this particular actor modeling for that name
That's one pairing that we are making
So compile now
After compiling
Now we see all the numbers of maps
That can be used for modeling pair
Click on plus
Every time you click it, it will be added
First one is, we had Hunet Cat
So once it gets that Hunet Cat name
For that name, let's assign
The BP_CatActor file
So here, since we added a data here
Let's use Get for Image Modeling Pair
Bring it here
And let's Find in that data
So the image we found
That Friendly Name added to that found image
Add it to this map
And then it will return some corresponding
Modeling Blueprint file like so
If there exists that name
In that Modeling Pair variable
So this modeling, let's go to Spawn Actor
To actually create it
First of all, connect the Exec Pin
Let's put it a little closer
And of the array of data images that we found
The first data
From there, the Friendly Name
Look for that added Friendly Name
And of the map that we made
When we search for it
First, with the names we created for this variable
It will return that modeling
And returning that modeling file or Blueprint file
Now it will create according to that file
When it does so
For Collision, let's go with Always Spawn
And then, for face detection and mask creation
It will also detect ground
As it always does for indicator spawning
But here, since it creates Tick in real time
And continue with every frame
If it keeps spawning
We know that it will overlap and keep spawning
So this time, to avoid that from the beginning
Let's save that spawned actor
As a variable at Promote Variables
So this new variable is called
spawnedActor
Let's name this variable that
This ensures that it doesn't spawn endlessly
It should check before it tries to spawn
So it brings in that value of the Spawned Actor
In numbers to see if there already exists one
So here, using Valid Node
It checks first if it has already been spawned
And it will continue to spawn based on that result
Now, connect this Valid node
And check if there already is an actor
In the variable Spawned Actor
If it doesn't have any
Or, if it is 'Is Not Valid'
It needs to spawn
Spawn and add to the varaible
Or, if it is already added
If there already is a value to the variable
If it already has spawned
Then, to that actor
Meaning, to Spawned Actor
Using Set Actor Transform
Its rotation and scale information is
All that needs to be updated
The image may move in between
And the camera may move as well
So the location may change all the time
So here, where do we spawn it
And where do we move it?
For that, we can bring it from
This Detected Image
Drag it here
Get Local to World Transform
So from the detected image
Convert Local to World coordinate
To get that Transform coordinate
So pass this location info on whenever we move it
Or whenever we spawn one
Then, it spawns and moves in place
Once it is already spawned
Let's also connect this Valid Exec Pin
This may look a little complicated
With the nodes intertwined like so
But you will have no trouble understanding it
Or, in case it gets confusing
Hover over a pin
Then it shows in a bold line
Where this pin is connected to
So it will be easier for you to check
Now, let's compile and save
Now, let's try packaging again
While it is being packaged
Let's open our image file in advance
Packages folder again
Go to Project, Packages folder
And this apk file
Ctrl+C to copy again
And on the phone, under Unreal App
Delete this file
And move it here again
Ctrl+V to paste
Close everything
And then at Installed Files
Let's install that apk file
Open to open the app
And select Image Type
So whenever we point to the imaeg
We see that modeling well
On top of the image like so
And you can try moving the image
Or moving the camera
We see how the modeling is moving along well
In fact, if we resize the image
We also see the scale of the modeling changing as well
Same with the rotation value2
Even if you rotate the screen like so
We see how the modeling is rotating
Along with the movement
Spawn Object That Corresponds to Each Image
But for now, we only have
One image added here
In order for use to actually use this
As in, to make it into a content
We may be required to add more than one images
Let's see how it works
When we are to add many images
First, we had another image that we imported
We had that sports car image
This requires a modeling for this car image
And let's use what Unreal provides
As a default Startup Content
Click on Add at Content Drawer
And on top, it says Add Feature or Content Pack
In order to use what Unreal provides as default
Let's select Add Feature or Content Pack
What we need here is
It's not a human modeling
We need some car modeling
So let's add Vehicle Project
Click on Dismiss
So this means that all the plugins related to cars
Could not be imported altogether
But we are not trying to use a plugin here
Nor are we trying to use its functions
We only need one thing
The modeling for this car
Since we don't need anything else
It doesn't matter if it says there is some error
It's because we simply don't use it
So, where is this modeling?
We have a folder called Vehicles here
At Vehicles, there's a folder called SportsCar
In there, as Skeletal Mesh
We have SKM SportsCar
That's the Skeletal Mesh we can use
Its Material looks nice enough
So we can just use this as it is here
So, at Blueprints folder
Just as we made our Cat actor
We will be creating a Blueprint class
For its parent class, let's select Actor
Let's say BP_CarActor
Same thing, for Component
Let's add a Skeletal Mesh Component
Add a Skeletal Mesh Component
Skeletal Mesh Asset
At Details, Skeletal Mesh Asset
The one we saw, the sports car
Let's add our SportsCar Mesh there
Of course, since it is a car, it is very big
Even bigger than 2 meters
So we need to resize it quite a bit
Since we need to add it to an image as well
So let's say 0.05, to really reduce its size
Compile and Save
For this, even as World
It is facing front as its modeling does
So we can use it as it is now
Save and close the settings window
Now, at Data Assets folder
We need to create a Hunet AR Session image
As well as a candidate image, of course
Right click, at Miscellaneous
Select Data Assets
First, candidate image
Let's start by creating AR Candidate IMage
Here, similar to the name before
Let's name this new file
ARImage_SportCar
Double click to open the settings
For Candidate TExture
What we called SCar, that image texture
Let's add that
Add in the sports car image
And its Friendly Name will be Hunet Car
Image size, width and height
Same as that cat, around 20cm
Let's add that
Orientation below has landscape and Portrait
But for this, since it is a square, it doesn't mean much
When the image is a rectangle, either longer to its width or length
If it is longer to the width, it is a landscape
And longer to the length is a portrait
Let's save and close the window
Next, Hunet AR Session Config Image
Open the image session file
Here, at Image Tracking, Candidate Image
As candidate image, we only added AR Image Cat
In order to add another
Let's click on the plus button above
And then, for candidate image
Let's add that sports car
Let's save and close the settings window
Now, back to the Blueprints
Go to Hunet Player Pawn again
And we need to change this detection part a little
Since we have more than one image
We cannot just get the 0th one
We need to repeat this array
And we need all the works to that image repeated
So this back part should be repeated per image
So let's push this away a little
Let's delete this Get part
We have to repeat the array now
So let's use a For Each Loop Node to
Repeat everything here
For the Exec Pin, connect it to this For Each Loop
And assign the detected image array here
Then this will repeat the array, from the 0th to the last
And every time it repeats
It will get that Detected Image
And read its Friendly Name there
If there is a value added
To that image modeling
For now, we only saved one thing, Hunet Cat
At our Image Modeling Pair variable
So here, let's click on the plus button
To add another pair
So for our Friendly Name
We added one called Hunet Car
And the modeling to be created is
Let's say this creates this BP_CarActor
Hunet Cat with Cat Actor
And Hunet Car with Car Actor
That's the pairs we have
So, whether this be the car or the cat
This will check each and get
The candidate name added to that image
And first see if it exists in the variable
If so, it will spawn the paired actor to return
Which is Actor Blueprint File
And now, adding on to the spawning
Let's also connect that target location
The spawning, the location moving, these are the same
But the problem is Spawned Actor
This will spawn something already spawned
Let's add to Spawned Actor
So that it does not duplicate
But this time, this actor may not be one
There can be more than one
Based on the number of images we added
So this single actor cannot contain all
For that, select Spawned Actor
And we need to change the data type of Spawned Actor
Spawned Actor, data type should not be a single Actor
Change it to Map
And it asks if your really want to change the data type
For that
Yes, we'd like to, so 'Change Variable Type'
Change it and close the window
Then this is now changed
So for Key, choose String
For Key, it is String
For Value, Actor
Since we are saving actual objects into the actor
Object Reference
It keeps asking if we are sure
For changing data type to Map
This is the inconvenient part of it
So Key is String
And the Value to be saved is Actor type
And for the actually created Actor Instance Type
Let's set Object Instance Type
Now, let's compile
For now, we get some error
Since all these parts such as Set
Have changed
Let's delete the node
So, whenever an actor is spawned
For that
We need to add it to Spawned Actor
So use Get at Spawned Actor
And using Add function
Let's save it using Add function
Let's save the spawned object
And for key value
For key name, we had it before
The name of the candidate image
So its nickname or Friendly Name goes here
Let's bring it down so that it is visually better
Now, let's double click and
Create the connecting node here
And make it go down and be seen like so
This is not mandatory but
Whenever all the lines get complicated
It may get very confusing
So let's deliver Friendly Name here
And that's the implementation
So whenever it is spawned
This spawned actor class with this name
It will get saved
And the next time, it found the existing one
And it will have that detected image with it
For that, let's delete this Valid node we had from before
This time, when checking if anything has already been spawned
Since the variable data type has changed to Map
Le's use Get at Spawned Actor
To see if there is anything like this included there
So let's use Contains
And add that key value name
So this will check first at Spawned Actor if it has any
And if it has already spawned it
Then Spawned Actor must have the key value there
So by checking for this key value here
If there isn't any
We will use a branch
And if not, this will be False
And if the key value is in there
This will be True
So if there is one already
Or if it is True
Then use Set Actor Transform to just change the location
Location and rotation values
And if this key isn't included in Spawned Actor
Then we need to add it anew
So for False, we use Spawn Actor
In order to spawn an actor
For the Branch node, let's connect it from this For Each Loop
Let's connect the Execution Pin like so
The nodes may look very complicated now
But it is very simple in essence
It just looks for an image from each part
It tries searching that image name at Spawned Actor
And if it finds one
Or, if there is no record of spawning
It spawns one
And after spawning, it uses that
Key name and the spawned actor instance
To save at Spawned Actor
And if there is record, when checked, of it having been spawned
Then it uses Actor Transform
To only move around its object value
And it can just manipulate that value
Here, we missed a step here
So, which actor's location value are we moving?
It must be something that has already been spawned
So at Spawned Actor, move up, Find
And the key from before
Friendly Name Key
We need to move that
Actual instance with that key
So let's reshape these lines
And then let's compile and save
And finally, let's try here
Let's build it again
And check for the last time
So let's try packaging once more
While it is doing the work, let's open the two images
Let's open these two images we used
So we have that cat image already open
And let's open that car image at the bottom
It's at Content, and I think at Textures
At Textures, open SCar image
The cat on the top
And let's put the car at the bottom
The packaging is now complete
Go to Packages, Ctrl+C the apk file
And delete the apk file in the phone
And paste here
Ctrl+V to copy and paste
Once it is installed, open
And open the app
Detection Type, Image Type
And as you see, we have two image detection
For the cat image, we have the cat modeling
And for the car, we have the car image
Along with the modeling
For both cat and car, the detection works well
And finally, there are some cautionary points
Regarding the images for detection
These images I have here
Have high recognition rates
High recognition rate means
If there is no transparency, it is easier to detect
With any transparent value, the recognition rate drops by a lot
So that's why I used block color for the background
In white so that there is no transparent value
And I also added an outline as you see here
Since the outline has a very bold color against the background
This outline distinguishes clearly between
The background and the content
And this will increase the recognition rate
And if the pattern inside is very intricate
If there are too many patterns inside
Then it is less likely to be recognized
In contrast, if the pattern and texture are rather simple
Then it is much easier to detect
Of course, for this, you will have to
Work with trial and error yourself
If possible, go with block image
As opposed to transparent image
And with simple background
With an outline around the content
This will make the image easier to detect
And it doesn't have to be an image file on the screen
Even when you print it out, make it into stickers
The same strategies work for the high recognition rate
Let's summarize what we learned today
First, we made our image textures to detect
Into AR Candidate files
And saved them as arrays at AR Session Config file
Next, we used a new data type called Map
Where we save data into Key and Value
And we also tried reading that data type
Finally, third, we put each image name as key
And found the corresponding actor
From the pre-made image database
And we also spawned that actor
On top of the detected image
AR Candidate Image and Real-World Image Recognition Implementing image recognition In the DataAssets folder, select [+Add]-[Miscellaneous]-[Data Assets]
After selecting ARSession Config, create HunetARSessionConfig_Image Specify Session Type World to use the rear camera
After selecting AR Candidate Image, create ARImage_cat Add cat.png file to Candidate Texture item
After adding array to Candidate Images, add ARImage_cat. Add cat.png file
When Switch on EDetection Type is Image Detection. Run HunetARSessionConfig_Image
If the ECurrent Detection value is selected as Image Detection, run ARTracked Image
Create 3D objects corresponding to recognized images Implementation of actor spawning function corresponding to image
Add cat_Idle.fbx and Texture.png files to the Mesh folder Connect Base Color to ult’s [Texture Sample] node
Select [Blueprints]-[Blueprint Class]-[Actor] and create BP_CatActor
Set the Scale value to (0.2, 0.2, 0.2) to match the recognized image size Set the Rotation value to (0, 0, -90) so that the modeling can face the front
Connect the [Get Frendly Name] node to the Get Detected image
After creating a variable called ImageModelingPair, change it to Map so that Key and Value data can be stored together
When you insert Get Friendly Name into Map, the corresponding blueprint file is returned
To prevent actors from being created overlapping, create the variable spawnedActor and connect the Spawned Actor and the [Is Valid] node
After creating Get Local to World Transform, connect [Set Actor Transform] and [SpawnActor] respectively
Create objects corresponding to each image Recognizing multiple images Delete the existing [Get] node and connect the [For Each Loop] node
Enter HunetCar as the second modeling pair value and assign the BP_CarActor file as the value
Convert the Variable Type of Spawned Actor to Map Change Key in Description to String
Change value in Description to [Actor]-[Object Reference]
Save the actual created object using the [Get spawnedActor]-[Add] function
The key value of [Add] is connected to [Get Friendly Name] [Get spawnedActor] - Connect [Contains] and [Get Friendly Name]
Add [Get spawnedActor]-[Contains]-[Branch] Connect [Branch] node to [For Each Loop]
If there is no created record, save the name and actual created instance after creating the Spawned Actor
If there is a created record, move the actual object value using Set Actor Transform
The End